Category: Power Query

In my last article, I showed you how to connect the quote endpoint from the Yahoo Finance API to Power BI. By leveraging the documentation and using copy/paste, we successfully connected to the API without a single line of code. In our review of the outcome we noted that our response only brought back data for a single stock.

While helpful, we know you probably want to refresh data for more than one stock. In fact, I happen to have a list of stocks that I am currently watching:

My next challenge is to take my original query and make it sustainable for however many stocks I have on my watch list.

Options are Plentiful

As with every Microsoft product, I have several different ways that I could manage this challenge. All have their benefits and drawbacks. Let’s look at them each and see which one would work best for my scenario.

Option 1 – Append Individual Queries

The first solution you might have considered is building a separate connection for each of the individual stocks we have in our list. Each query would be individually created and then appended into a master list so that way it stays organized. This solution would work and be effective, but there are drawbacks.

First of all, the biggest draw back is that I have to manually create this query each and every time I want to add a new stock. This does not take a ton of time, but it would require a few minutes of effort each and every time. Not a big deal if you only need to add one or two stocks to the model but it would add up with 20-30 stocks.

The other drawback is that you will have to continue managing the data model. If the list of stocks is dynamic, that means you either need to create a single model with every single stock in it or continue to manage it over time. This is not really sustainable as new stocks enter the market on a regular basis and requires manual maintenance.

Option 2 – Go Back to the Documentation

Sometimes you have some options when it comes to your endpoints for expanding your query. If we go back to look at our documentation for the quote endpoint, you will see that we can specify up to 10 stock symbols in our query.

Perfect! So instead of having to write out 25 separate queries, I can condense it all into three. That is much more manageable and efficient! Well – almost more efficient…

Remember what I stated above – because we are still specifying the stock symbols for each query we still have to manually intervene every time we make a change to our stock list. This could get difficult especially as we go to maintain our list over time as we might need to remove stocks later. As a result, we will spend time searching for which query to resolve.

Option 3 – Scale Out our Original Query as a Function

The last option we have is to make our current query scalable. How do we make a query scalable? By converting it into a function.

Functions allow us to reuse that query over and over again to simplify our refresh. Acting similar to our first option, we can apply a function to a list of stock symbols and quickly bring back the quote information. Now instead of managing the queries, we are just managing a list of stock symbols. As symbols are added and removed, our data model adjusts accordingly. The result allows us to automate refreshes without having to adjust the data model in the future.

Converting a Query to a Function

Remember how I said you might need some minimal coding? That time has come. We will need to do something you may have never done before – open the advanced editor window in Power Query.

Before we do that, I do want to rename the current query to “Get-StockInfo” so it is a little easier to read. Just update the name on the query settings bar on the right. I like using the word get in front to help quickly identify it as a function versus a query.

Once you have done that, click on Advanced Editor in the home ribbon and strap in for the ride!

Step 1 – Convert the Query to a Function

Our goal is to make this function flexible for any stock symbol. That means we need to require a stock symbol to be called every time the function is leveraged.

When you open the code editor, the first line has the single word “let”. We are going to insert a row above that and put a pair of parentheses followed by an equals sign and greater than sign. This effectively makes the conversion from a query to a function.

()=>
let
    Source = Json.Document(Web.Contents("https://yfapi.net/v6/finance/quote?region=US&lang=en&symbols=MSFT", [Headers=[Accept="application/json", #"X-API-KEY"="bHjSKbiRtr4cOzmOopdrI5TUKX5kMKXU6weUteeL"]])),
    #"Converted to Table" = Table.FromRecords({Source}),
    #"Expanded quoteResponse" = Table.ExpandRecordColumn(#"Converted to Table", "quoteResponse", {"result", "error"}, {"quoteResponse.result", "quoteResponse.error"}),
    #"Expanded quoteResponse.result" = Table.ExpandListColumn(#"Expanded quoteResponse", "quoteResponse.result"),
    #"Expanded quoteResponse.result1" = Table.ExpandRecordColumn(#"Expanded quoteResponse.result", "quoteResponse.result", {"language", "region", "quoteType", "quoteSourceName", "triggerable", "currency", "firstTradeDateMilliseconds", "postMarketChangePercent", "postMarketTime", "postMarketPrice", "postMarketChange", "regularMarketChange", "regularMarketChangePercent", "regularMarketTime", "regularMarketPrice", "regularMarketDayHigh", "regularMarketDayRange", "regularMarketDayLow", "regularMarketVolume", "regularMarketPreviousClose", "bid", "ask", "bidSize", "askSize", "fullExchangeName", "financialCurrency", "regularMarketOpen", "averageDailyVolume3Month", "averageDailyVolume10Day", "fiftyTwoWeekLowChange", "fiftyTwoWeekLowChangePercent", "fiftyTwoWeekRange", "fiftyTwoWeekHighChange", "fiftyTwoWeekHighChangePercent", "fiftyTwoWeekLow", "fiftyTwoWeekHigh", "dividendDate", "marketState", "earningsTimestamp", "earningsTimestampStart", "earningsTimestampEnd", "trailingAnnualDividendRate", "trailingPE", "trailingAnnualDividendYield", "epsTrailingTwelveMonths", "epsForward", "epsCurrentYear", "priceEpsCurrentYear", "sharesOutstanding", "bookValue", "fiftyDayAverage", "fiftyDayAverageChange", "fiftyDayAverageChangePercent", "twoHundredDayAverage", "twoHundredDayAverageChange", "twoHundredDayAverageChangePercent", "marketCap", "forwardPE", "priceToBook", "sourceInterval", "exchangeDataDelayedBy", "pageViewGrowthWeekly", "averageAnalystRating", "tradeable", "priceHint", "exchange", "shortName", "longName", "messageBoardId", "exchangeTimezoneName", "exchangeTimezoneShortName", "gmtOffSetMilliseconds", "market", "esgPopulated", "displayName", "symbol"}, {"language", "region", "quoteType", "quoteSourceName", "triggerable", "currency", "firstTradeDateMilliseconds", "postMarketChangePercent", "postMarketTime", "postMarketPrice", "postMarketChange", "regularMarketChange", "regularMarketChangePercent", "regularMarketTime", "regularMarketPrice", "regularMarketDayHigh", "regularMarketDayRange", "regularMarketDayLow", "regularMarketVolume", "regularMarketPreviousClose", "bid", "ask", "bidSize", "askSize", "fullExchangeName", "financialCurrency", "regularMarketOpen", "averageDailyVolume3Month", "averageDailyVolume10Day", "fiftyTwoWeekLowChange", "fiftyTwoWeekLowChangePercent", "fiftyTwoWeekRange", "fiftyTwoWeekHighChange", "fiftyTwoWeekHighChangePercent", "fiftyTwoWeekLow", "fiftyTwoWeekHigh", "dividendDate", "marketState", "earningsTimestamp", "earningsTimestampStart", "earningsTimestampEnd", "trailingAnnualDividendRate", "trailingPE", "trailingAnnualDividendYield", "epsTrailingTwelveMonths", "epsForward", "epsCurrentYear", "priceEpsCurrentYear", "sharesOutstanding", "bookValue", "fiftyDayAverage", "fiftyDayAverageChange", "fiftyDayAverageChangePercent", "twoHundredDayAverage", "twoHundredDayAverageChange", "twoHundredDayAverageChangePercent", "marketCap", "forwardPE", "priceToBook", "sourceInterval", "exchangeDataDelayedBy", "pageViewGrowthWeekly", "averageAnalystRating", "tradeable", "priceHint", "exchange", "shortName", "longName", "messageBoardId", "exchangeTimezoneName", "exchangeTimezoneShortName", "gmtOffSetMilliseconds", "market", "esgPopulated", "displayName", "symbol"})
in
    #"Expanded quoteResponse.result1"

However, we have not done anything to handle the requirement to pass a stock symbol when the function is called. To require a stock symbol be called, we will simply add a variable called StockSymbol and specify that it be passed in a text format. This will require a stock symbol be passed any time you leverage this function.

(StockSymbol as text)=>
let
    Source = Json.Document(Web.Contents("https://yfapi.net/v6/finance/quote?region=US&lang=en&symbols=MSFT", [Headers=[Accept="application/json", #"X-API-KEY"="bHjSKbiRtr4cOzmOopdrI5TUKX5kMKXU6weUteeL"]])),
    #"Converted to Table" = Table.FromRecords({Source}),
    #"Expanded quoteResponse" = Table.ExpandRecordColumn(#"Converted to Table", "quoteResponse", {"result", "error"}, {"quoteResponse.result", "quoteResponse.error"}),
    #"Expanded quoteResponse.result" = Table.ExpandListColumn(#"Expanded quoteResponse", "quoteResponse.result"),
    #"Expanded quoteResponse.result1" = Table.ExpandRecordColumn(#"Expanded quoteResponse.result", "quoteResponse.result", {"language", "region", "quoteType", "quoteSourceName", "triggerable", "currency", "firstTradeDateMilliseconds", "postMarketChangePercent", "postMarketTime", "postMarketPrice", "postMarketChange", "regularMarketChange", "regularMarketChangePercent", "regularMarketTime", "regularMarketPrice", "regularMarketDayHigh", "regularMarketDayRange", "regularMarketDayLow", "regularMarketVolume", "regularMarketPreviousClose", "bid", "ask", "bidSize", "askSize", "fullExchangeName", "financialCurrency", "regularMarketOpen", "averageDailyVolume3Month", "averageDailyVolume10Day", "fiftyTwoWeekLowChange", "fiftyTwoWeekLowChangePercent", "fiftyTwoWeekRange", "fiftyTwoWeekHighChange", "fiftyTwoWeekHighChangePercent", "fiftyTwoWeekLow", "fiftyTwoWeekHigh", "dividendDate", "marketState", "earningsTimestamp", "earningsTimestampStart", "earningsTimestampEnd", "trailingAnnualDividendRate", "trailingPE", "trailingAnnualDividendYield", "epsTrailingTwelveMonths", "epsForward", "epsCurrentYear", "priceEpsCurrentYear", "sharesOutstanding", "bookValue", "fiftyDayAverage", "fiftyDayAverageChange", "fiftyDayAverageChangePercent", "twoHundredDayAverage", "twoHundredDayAverageChange", "twoHundredDayAverageChangePercent", "marketCap", "forwardPE", "priceToBook", "sourceInterval", "exchangeDataDelayedBy", "pageViewGrowthWeekly", "averageAnalystRating", "tradeable", "priceHint", "exchange", "shortName", "longName", "messageBoardId", "exchangeTimezoneName", "exchangeTimezoneShortName", "gmtOffSetMilliseconds", "market", "esgPopulated", "displayName", "symbol"}, {"language", "region", "quoteType", "quoteSourceName", "triggerable", "currency", "firstTradeDateMilliseconds", "postMarketChangePercent", "postMarketTime", "postMarketPrice", "postMarketChange", "regularMarketChange", "regularMarketChangePercent", "regularMarketTime", "regularMarketPrice", "regularMarketDayHigh", "regularMarketDayRange", "regularMarketDayLow", "regularMarketVolume", "regularMarketPreviousClose", "bid", "ask", "bidSize", "askSize", "fullExchangeName", "financialCurrency", "regularMarketOpen", "averageDailyVolume3Month", "averageDailyVolume10Day", "fiftyTwoWeekLowChange", "fiftyTwoWeekLowChangePercent", "fiftyTwoWeekRange", "fiftyTwoWeekHighChange", "fiftyTwoWeekHighChangePercent", "fiftyTwoWeekLow", "fiftyTwoWeekHigh", "dividendDate", "marketState", "earningsTimestamp", "earningsTimestampStart", "earningsTimestampEnd", "trailingAnnualDividendRate", "trailingPE", "trailingAnnualDividendYield", "epsTrailingTwelveMonths", "epsForward", "epsCurrentYear", "priceEpsCurrentYear", "sharesOutstanding", "bookValue", "fiftyDayAverage", "fiftyDayAverageChange", "fiftyDayAverageChangePercent", "twoHundredDayAverage", "twoHundredDayAverageChange", "twoHundredDayAverageChangePercent", "marketCap", "forwardPE", "priceToBook", "sourceInterval", "exchangeDataDelayedBy", "pageViewGrowthWeekly", "averageAnalystRating", "tradeable", "priceHint", "exchange", "shortName", "longName", "messageBoardId", "exchangeTimezoneName", "exchangeTimezoneShortName", "gmtOffSetMilliseconds", "market", "esgPopulated", "displayName", "symbol"})
in
    #"Expanded quoteResponse.result1"

Step 2 – Make the Connection String Dynamic

Now that we converted the query to a function and required a stock symbol be passed, it is time for us to use it dynamically with the connection string.

On line 3, you see our endpoint url and parameters that specify MSFT as our designated stock. We need to replace MSFT with our new variable. We will simply delete the current symbol, add an ampersand (&) outside of the quotes, and specify StockSymbol to concatenate the values together.

(StockSymbol as text)=>
let
    Source = Json.Document(Web.Contents("https://yfapi.net/v6/finance/quote?region=US&lang=en&symbols=" & StockSymbol, [Headers=[Accept="application/json", #"X-API-KEY"="bHjSKbiRtr4cOzmOopdrI5TUKX5kMKXU6weUteeL"]])),
    #"Converted to Table" = Table.FromRecords({Source}),
    #"Expanded quoteResponse" = Table.ExpandRecordColumn(#"Converted to Table", "quoteResponse", {"result", "error"}, {"quoteResponse.result", "quoteResponse.error"}),
    #"Expanded quoteResponse.result" = Table.ExpandListColumn(#"Expanded quoteResponse", "quoteResponse.result"),
    #"Expanded quoteResponse.result1" = Table.ExpandRecordColumn(#"Expanded quoteResponse.result", "quoteResponse.result", {"language", "region", "quoteType", "quoteSourceName", "triggerable", "currency", "firstTradeDateMilliseconds", "postMarketChangePercent", "postMarketTime", "postMarketPrice", "postMarketChange", "regularMarketChange", "regularMarketChangePercent", "regularMarketTime", "regularMarketPrice", "regularMarketDayHigh", "regularMarketDayRange", "regularMarketDayLow", "regularMarketVolume", "regularMarketPreviousClose", "bid", "ask", "bidSize", "askSize", "fullExchangeName", "financialCurrency", "regularMarketOpen", "averageDailyVolume3Month", "averageDailyVolume10Day", "fiftyTwoWeekLowChange", "fiftyTwoWeekLowChangePercent", "fiftyTwoWeekRange", "fiftyTwoWeekHighChange", "fiftyTwoWeekHighChangePercent", "fiftyTwoWeekLow", "fiftyTwoWeekHigh", "dividendDate", "marketState", "earningsTimestamp", "earningsTimestampStart", "earningsTimestampEnd", "trailingAnnualDividendRate", "trailingPE", "trailingAnnualDividendYield", "epsTrailingTwelveMonths", "epsForward", "epsCurrentYear", "priceEpsCurrentYear", "sharesOutstanding", "bookValue", "fiftyDayAverage", "fiftyDayAverageChange", "fiftyDayAverageChangePercent", "twoHundredDayAverage", "twoHundredDayAverageChange", "twoHundredDayAverageChangePercent", "marketCap", "forwardPE", "priceToBook", "sourceInterval", "exchangeDataDelayedBy", "pageViewGrowthWeekly", "averageAnalystRating", "tradeable", "priceHint", "exchange", "shortName", "longName", "messageBoardId", "exchangeTimezoneName", "exchangeTimezoneShortName", "gmtOffSetMilliseconds", "market", "esgPopulated", "displayName", "symbol"}, {"language", "region", "quoteType", "quoteSourceName", "triggerable", "currency", "firstTradeDateMilliseconds", "postMarketChangePercent", "postMarketTime", "postMarketPrice", "postMarketChange", "regularMarketChange", "regularMarketChangePercent", "regularMarketTime", "regularMarketPrice", "regularMarketDayHigh", "regularMarketDayRange", "regularMarketDayLow", "regularMarketVolume", "regularMarketPreviousClose", "bid", "ask", "bidSize", "askSize", "fullExchangeName", "financialCurrency", "regularMarketOpen", "averageDailyVolume3Month", "averageDailyVolume10Day", "fiftyTwoWeekLowChange", "fiftyTwoWeekLowChangePercent", "fiftyTwoWeekRange", "fiftyTwoWeekHighChange", "fiftyTwoWeekHighChangePercent", "fiftyTwoWeekLow", "fiftyTwoWeekHigh", "dividendDate", "marketState", "earningsTimestamp", "earningsTimestampStart", "earningsTimestampEnd", "trailingAnnualDividendRate", "trailingPE", "trailingAnnualDividendYield", "epsTrailingTwelveMonths", "epsForward", "epsCurrentYear", "priceEpsCurrentYear", "sharesOutstanding", "bookValue", "fiftyDayAverage", "fiftyDayAverageChange", "fiftyDayAverageChangePercent", "twoHundredDayAverage", "twoHundredDayAverageChange", "twoHundredDayAverageChangePercent", "marketCap", "forwardPE", "priceToBook", "sourceInterval", "exchangeDataDelayedBy", "pageViewGrowthWeekly", "averageAnalystRating", "tradeable", "priceHint", "exchange", "shortName", "longName", "messageBoardId", "exchangeTimezoneName", "exchangeTimezoneShortName", "gmtOffSetMilliseconds", "market", "esgPopulated", "displayName", "symbol"})
in
    #"Expanded quoteResponse.result1"

With the dynamic variable placed in the connection string, the function will now pull stock quote information based upon the symbols we have in our list.

Step 3 – Resolve Connection String Errors

If you clicked done and used this function in your report, everything would be fine inside of Power BI Desktop. However, you are going to run into an issue when you publish to the Power BI Service. Because we are using a dynamic parameter, the cloud service is not sure how to handle authentication. The result is an error in the Power BI Service and a report that will not refresh.’

To resolve this issue, we want to split our connection string to resolve this issue. We will essentially split it into two different sections. The first section will serve as the connection string and help with our authentication. The second half will be treated as a dynamic portion of the query and has no bearing on the authentication process.

In short, all we are going to do is move a portion of that connection string to prevent any errors from happening.

(StockSymbol as text)=>
let
    Source = Json.Document(Web.Contents("https://yfapi.net/v6/finance/", [RelativePath="quote?region=US&lang=en&symbols=" & StockSymbol, Headers=[Accept="application/json", #"X-API-KEY"="bHjSKbiRtr4cOzmOopdrI5TUKX5kMKXU6weUteeL"]])),
    #"Converted to Table" = Table.FromRecords({Source}),
    #"Expanded quoteResponse" = Table.ExpandRecordColumn(#"Converted to Table", "quoteResponse", {"result", "error"}, {"quoteResponse.result", "quoteResponse.error"}),
    #"Expanded quoteResponse.result" = Table.ExpandListColumn(#"Expanded quoteResponse", "quoteResponse.result"),
    #"Expanded quoteResponse.result1" = Table.ExpandRecordColumn(#"Expanded quoteResponse.result", "quoteResponse.result", {"language", "region", "quoteType", "quoteSourceName", "triggerable", "currency", "firstTradeDateMilliseconds", "postMarketChangePercent", "postMarketTime", "postMarketPrice", "postMarketChange", "regularMarketChange", "regularMarketChangePercent", "regularMarketTime", "regularMarketPrice", "regularMarketDayHigh", "regularMarketDayRange", "regularMarketDayLow", "regularMarketVolume", "regularMarketPreviousClose", "bid", "ask", "bidSize", "askSize", "fullExchangeName", "financialCurrency", "regularMarketOpen", "averageDailyVolume3Month", "averageDailyVolume10Day", "fiftyTwoWeekLowChange", "fiftyTwoWeekLowChangePercent", "fiftyTwoWeekRange", "fiftyTwoWeekHighChange", "fiftyTwoWeekHighChangePercent", "fiftyTwoWeekLow", "fiftyTwoWeekHigh", "dividendDate", "marketState", "earningsTimestamp", "earningsTimestampStart", "earningsTimestampEnd", "trailingAnnualDividendRate", "trailingPE", "trailingAnnualDividendYield", "epsTrailingTwelveMonths", "epsForward", "epsCurrentYear", "priceEpsCurrentYear", "sharesOutstanding", "bookValue", "fiftyDayAverage", "fiftyDayAverageChange", "fiftyDayAverageChangePercent", "twoHundredDayAverage", "twoHundredDayAverageChange", "twoHundredDayAverageChangePercent", "marketCap", "forwardPE", "priceToBook", "sourceInterval", "exchangeDataDelayedBy", "pageViewGrowthWeekly", "averageAnalystRating", "tradeable", "priceHint", "exchange", "shortName", "longName", "messageBoardId", "exchangeTimezoneName", "exchangeTimezoneShortName", "gmtOffSetMilliseconds", "market", "esgPopulated", "displayName", "symbol"}, {"language", "region", "quoteType", "quoteSourceName", "triggerable", "currency", "firstTradeDateMilliseconds", "postMarketChangePercent", "postMarketTime", "postMarketPrice", "postMarketChange", "regularMarketChange", "regularMarketChangePercent", "regularMarketTime", "regularMarketPrice", "regularMarketDayHigh", "regularMarketDayRange", "regularMarketDayLow", "regularMarketVolume", "regularMarketPreviousClose", "bid", "ask", "bidSize", "askSize", "fullExchangeName", "financialCurrency", "regularMarketOpen", "averageDailyVolume3Month", "averageDailyVolume10Day", "fiftyTwoWeekLowChange", "fiftyTwoWeekLowChangePercent", "fiftyTwoWeekRange", "fiftyTwoWeekHighChange", "fiftyTwoWeekHighChangePercent", "fiftyTwoWeekLow", "fiftyTwoWeekHigh", "dividendDate", "marketState", "earningsTimestamp", "earningsTimestampStart", "earningsTimestampEnd", "trailingAnnualDividendRate", "trailingPE", "trailingAnnualDividendYield", "epsTrailingTwelveMonths", "epsForward", "epsCurrentYear", "priceEpsCurrentYear", "sharesOutstanding", "bookValue", "fiftyDayAverage", "fiftyDayAverageChange", "fiftyDayAverageChangePercent", "twoHundredDayAverage", "twoHundredDayAverageChange", "twoHundredDayAverageChangePercent", "marketCap", "forwardPE", "priceToBook", "sourceInterval", "exchangeDataDelayedBy", "pageViewGrowthWeekly", "averageAnalystRating", "tradeable", "priceHint", "exchange", "shortName", "longName", "messageBoardId", "exchangeTimezoneName", "exchangeTimezoneShortName", "gmtOffSetMilliseconds", "market", "esgPopulated", "displayName", "symbol"})
in
    #"Expanded quoteResponse.result1"

This is a subtle but important move. By placing the dynamic content inside the relative path, we have eliminated confusion during the authentication process. Power BI will authenticate on the connection string in place and then concatenate the relative path to complete the request. You only need to do this when your queries are dynamic, but it is an important step in the process!

Step 4 – Test the Function

After clicking done, your view in Power Query will have changed. Note that the query list now shows an “fx” symbol for function and you no longer have a tabular list of data. In fact, you have a place to enter a stock symbol. This allows us to quickly test to see if our function is working. Go ahead and enter MSFT and click invoke to see it work.

After you invoke the function, you will be taken to a new query called “Invoked Function” and can see the stock data for Microsoft. That was just to test and see if it works. We can go ahead and delete that query because will not need it again.

Applying the Function

With a working function in hand, it is time to apply it to our data model. I already have a list of stocks in Excel, so I will import that list to Power BI. If your stock list is somewhere else, that is perfectly fine. Just get that list of symbols inside of Power BI.

Once imported, go to your “Add Column” tab and select “Invoke Custom Function”. Select your function from the list. You will notice that the StockSymbol variable will appear. We need to specify which column of values will be passed to that variable. Select the symbol column and click OK.

At this point, you may be prompted about data privacy. If you do, just select ignore privacy levels for now as this is a demonstration.

You should have a new column with the word table highlighted inside of it. We can go ahead and expand those values and confirm the current stock quote info is available for each different symbol.

And just like that you created your own function and applied it in Power Query!

Next Steps

With a working function, what is left to discuss? As I stated at the beginning of this series – each API is like a snowflake. They all are unique and have their own challenges. This is a single example of how to connect a REST API to Power BI. There are challenges and other considerations you must balance when creating your reports. In my next post, I will dive into challenges you will see when leveraging other APIs inside of Power BI.

Until then, was this your first time creating a function? Did you find it challenging? Do you have other applications for a function inside of Power BI? Tell me in the comments below!

In my last article, I discussed the basics around REST APIs. We did not do anything with Power BI. While I did not get to the real purpose of this series, it was important to establish the basics. Using the elements we discussed, you can copy and paste them into Power BI to connect to your API.

For this post, I am using the Yahoo Finance API to demonstrate how this works. I do not receive any kick back for sharing this API. They have a free tier and it is a great API to use for beginners. If you want to try this on your own, you can follow along with this article.

RTFM – Read the Fine Manual

Before you do anything with APIs you must review the documentation. I worked with an API a few years ago that required a unique API key for each individual endpoint. If I had spent some time reviewing the documentation, I would have prevented hours of frustration as I tried to connect to various endpoints. If you are like me, you might find this frustrating. However, I find reading the documentation far less frustrating than constantly struggling to get an endpoint to work inside of Power BI.

Identifying Usable Endpoints

The first time I review documentation I am sorting out which endpoints are useable or not. I need to find out which API endpoints that will work with Power BI. I opened the documentation and was excited about what I saw…

At a quick glace, all of the endpoints are using the GET connection method. As a result that means every single endpoint can be used in Power BI. That is great to see!

Which Endpoint is for Me?

Knowing what you are looking for is key when reviewing documentation. I want to find stock price information and compare prices. Quality documentation will have one of two things to help you with this process:

• A data dictionary that clearly defines each column in the query
• A same response from the query

This documentation brings back a sample. As expected, I can find which endpoint meets my needs. As an example, I am interested in stock prices. According to the documentation, I can use the quote endpoint to accomplish my goal.

Setting the Parameters

Now that we have the connection method and endpoint, we need to establish what parameters are needed. Sometimes parameters are identified but not clearly documented. As a result, you might get frustrated because you cannot apply parameters to your query. This API is much easier to understand because of the clearly labeled parameters with sample inputs.

As you can see, the documentation provides sample values for each parameter so you know how to leverage it. This results in far less frustration because you know what values are accepted for each parameter.

Since this API has an option to try it out, I have updated the parameters to pull back Microsoft’s stock information. This allowed me to validate if my parameters were correct and saved me a ton of headaches. Since it worked, you can see that it came back with a response.

As a result of the test, there was another valuable bonus. The documentation assembled my connection string by putting the endpoint url and parameters together into a single connection string. How convenient!

Header Information

Up to this point the Yahoo Finance API documentation has explicitly called out each of the REST API elements. However, the headers in this endpoint are not as easily identified. We have to dig a little deeper to find them.

You probably missed the box above the connection string labeled Curl. Curl is a library used to call APIs if you are developing an application. While we are not developing an application, this box does help us identify our headers. If you notice, there are two lines with “-H” at the beginning. These are the header values we have been looking for!

Double Check

At this stage, we have identified all the elements we need for our connection in Power BI. We have the connection method, endpoint url, parameters, and headers. Remember that we are not using a POST connection method, so we do not need a body for this request. Now we just need to plug it into Power BI!

Connecting to our REST API (Finally!)

With all of our elements in hand, we can open up Power BI and start building our connection to the API.

Step 1 – Select the Web Connector

Open Power BI, select “Get Data”, and choose the Web connector:

Step 2 – Copy/Paste Elements into Power BI

First things first – switch to the advanced mode for the Web Connector so you can add all of the required details.

You will notice that there is no option to select your connection method in this view. Power BI defaults this to GET, so you do not need to configure anything. One less thing to worry about!

Go back to the documentation and copy your connection string. If your documentation does not assemble this for you, you might need to do this by hand. Once assembled, paste it into the URL Parts box in the web connector.

Next, we can add the headers. This endpoint has two – Accept and X-API-KEY. You will only see one set of boxes at the bottom. You can simply click “Add header” to accommodate for the second header.

For the Accept header, we can select it from the drop down on the left as it is a commonly used header. On the right, copy and paste the value found in the API documentation.

For the X-API-KEY header, you can enter your own value in the drop down or copy/paste it from your documentation. You can copy your API key from the documentation as well into the box on the right.

By bridging the documentation with the web connector, we have successfully filled in all of the blanks for the connection method, endpoint url, parameters, and headers. With everything in place, we can click okay and connect!

Step 3 – Authentication

Since this is the first time we are using this connection, we are prompted to select an authentication method. If you are using the Yahoo Finance API, you can easily select Anonymous and click connect as our API key handles our authentication.

On the left, you can see there are other authentication methods. Sometimes your connection method will require a username and password. I have experienced this with ServiceNow and Honeywell APIs. If you are adventurous and want to try using the Microsoft Graph API, you will need to use your organizational account. You will need to review your documentation to determine if you need to use other forms of authentication to use an endpoint.

Transformation Time

By following these steps, you should now see your stock data inside of Power Query. All of the information you saw in the sample response should be in a tabular format and ready to go. For an easier view, I made some adjustments so you could easily compare the values:

With this query in place, you can quickly refresh the data and will always have up to date stock quote information for Microsoft.

Next Steps

With everything connected, you probably think we are finished. If there was only one stock we cared about, we would be in good shape. However, investors are looking at several different stocks each and every day and this query only pulled one. In my next article, I will show you how to scale this query so whether you have 10 or 1,000, you will have a maintenance free report that always refreshes.

Until then, did you follow along with the Yahoo Finance API? Were you able to get some valid data? Tell me in the comments below!

We have all been there once before. We are using a 3rd party tool and we want to extract data from it. You might have an option to export the data to a CSV or Excel file, but that requires you to be involved and can result in errors.

Next you might reach out to the support team and they send back documentation for their APIs. They think you are a programmer and probably know how to use them. But in a world full of citizen developers not everyone knows how to use these APIs to bring data into Power BI.

This series aims to demystify the process and start getting that data into Power BI and eliminate those data exports. And believe it or not, it will require a minimal amount of code to make it happen!

What is REST?

REST APIs (also known as RESTful APIs) is a specific architecture of an Application Program Interface (API). Known as the Representational State Transfer (REST), it has a standard set of rules that are widely used in the programming world. It scales well and has become a commonly accepted API that is stable. REST APIs fill a specific need to simplify integration of data between different applications.

API requests are fairly straight forward. It starts with a request from you (the client) to an API endpoint. After being authenticated, the request continues on to a database with the data you need. That data is returned back to the endpoint where it is formatted and then passed back to the client.

For many, this flow makes a ton of sense. However, if you are not from a programming background, this can be a little intimidating. When you read API documentation, it is often full of code snippets and it can be difficult for you to get started. Sorting through these snippets and figuring out how they work can be complicated and time consuming.

But here is the good news – Power BI can handle these API requests often with no code required! The trick is understanding how REST APIs are structured and then plugging the information from the documentation into the right place.

Anatomy of REST APIs

So if we can take the documentation and plug it into Power BI, it is important to understand how REST APIs are constructed. Once we understand the anatomy of an API, it becomes easier to connect via Power BI. There are five common elements that are leveraged with REST APIs:

• Method
• Endpoint URL
• Parameters
• Headers
• Body

You will always need a Method and Endpoint URL when using REST. However, the other elements will vary based off of the documentation. You will have to read your documentation carefully to make sure you have everything you need.

To better understand these elements, let’s take a look at them individually to understand what they do and how they work.

Method

There are four methods that are used with REST APIs: Get, Post, Put, and Delete. Methods help instruct what is happening when the API is called. For example, if you called a Delete method, you would be deleting content on the server.

To make our lives easier, the Get method is the most common one we will use and is the easiest to implement. Post can be used sometimes, but it requires a little more effort and coding in Power Query to make it work. You will never use Put or Delete in Power BI.

Endpoint URL

The Endpoint URL determines which specific API we are calling. There will be multiple APIs available, so it is important to determine which one you need. In reviewing your documentation, you should be able to determine which endpoint contains the data you need to query.

Parameters

Parameters filter the results of your request. Notice the “?” that separates the connection string. Before it is the Endpoint URL while after are the parameters. Multiple parameters are separated by ampersands (&) and strung together.

Parameters speed up requests as they reduce the amount of data requested. Often optional, but sometimes required. The API documentation will specify if any parameters are required.

Headers

Headers are designed to pass information about your request. The most common header you will need is an API key or token but you may run into others. Another common header you might need to identify is the Content-Type to specify how we want to see our response.

Body

Since we are using the Get method primarily in Power BI, you will not need the body. However, it is worth mentioning it in case you use a Post method in the future. The body is another set of parameters to assist with the request.

Documentation is your friend

If you are like me then you hate reading directions. I love working through things on my own and discovering. But after many lost hours, I have learned to take a few minutes to read the documentation that comes with an API first. It takes time, but there are several reasons to take a few minutes and understand how the API functions.

Reason #1 – Each API is like a snowflake

Every API is unique and has their own requirements. You will find some that require certain parameters to be declared. Others might need a token. I even worked with one that required a different API key for each endpoint! Understanding the components of our API will reduce headaches as we go to use them.

Reason #2 – Ensure you are receiving the right data

The best documentation is paired with a dictionary to help identify the data being returned. You may need two or three APIs to get all the information you need. Taking a few minutes to understand what data is available will help you build a plan as you build your connections.

Reason #3 – Try it out

Often APIs have an option to try them out. You can select an endpoint, fill out the parameters, and then submit the request. It brings back a sample of the data and even will assemble the connection string and headers for you. It also helps identify any required parameters and diagnose issues before you move into Power BI.

Next Steps

Now that we understand the basics around REST APIs, the next step is to transfer what we have learned into Power BI. You might be wondering why I took the time to understand the basics of REST APIs. As I mentioned before, many of these APIs can be used with no code. With a strong understanding of how they are built, we can copy/paste these elements into Power BI and start querying data. In my next post, I will show you how to connect to the Yahoo Finance API to query stock quotes in real time.

Until then, have you used any APIs in Power BI? Do you have any favorites that you like to use? Tell me in the comments below!