Using Python and the Metering API to Produce Custom Cost Reports
February 28, 2020 | 9 minute readIn a previous blog post, I introduced the Oracle Cloud Metering API. The logical next step is to wrap some Python functions around the API calls and use them to get meaningful reports for a tenancy. This is what I’ll discuss here for a sample use case.
For the sake of this example, let’s assume we’re interested in cost by compartment. This could easily be modified to use any other type of cost tracking tags. But since compartments are already prepopulated and all OCI resources are associated with exactly one, this is a good start. So let’s say we want a report that, for a given month, shows the cost accumulated under each top level compartment, including all their subcompartments.
Using the API call already discussed in the previous post, here’s a sample function to query for the cost of a single compartment. It will take start and end time as arguments, along with the compartment ID we’re looking for:
def get_compartment_charges(meteringid, start_time, end_time, compartment):
# returns a bill for the given compartment.
username=meteringid['username']
password=meteringid['password']
idcs_guid=meteringid['idcs_guid']
domain=meteringid['domain']
compartmentbill=collections.defaultdict(dict)
url_params = {
'startTime': start_time.isoformat() + '.000',
'endTime': end_time.isoformat() + '.000',
'usageType': 'TOTAL',
'computeTypeEnabled': 'Y'
}
url_params.update( {'tags': 'ORCL:OCICompartment=' + compartment.id } )
resp = requests.get(
'https://itra.oraclecloud.com/metering/api/v1/usagecost/'
+ domain + '/tagged',
auth=(username, password),
headers={'X-ID-TENANT-NAME': idcs_guid , 'Accept-Encoding':'*' },
params=url_params
)
if resp.status_code != 200:
# This means something went wrong.
print('Error in GET: {}'.format(resp.status_code), file=sys.stderr)
raise ExceptionThis first part will build the API call, post it and collect the result in resp. Next, we’ll iterate over the items in this JSON structure to build our bill. If you want to dig deeper into the structure of the data returned, run this in a Python debugger and look at the live object as it’s returned by the call.
for item in resp.json()['items']:
itemcost=0
service=item['serviceName']
resource=item['resourceName']
for cost in item['costs']:
itemcost += cost['computedAmount']
try:
compartmentbill[service][resource]+=itemcost
except KeyError:
compartmentbill[service][resource]=itemcost
return compartmentbillThis will return a dict that looks like this (for the single service COMPUTEBAREMETAL):
{
"COMPUTEBAREMETAL": {
"PIC_COMPUTE_OUTBOUND_DATA_TRANSFER": 0,
"PIC_COMPUTE_STANDARD_E2": 45.3,
"PIC_LB_SMALL": 8.067375,
"PIC_OBJECT_STORAGE_TIERED": 1.9079301671751942e-05,
"PIC_STANDARD_PERFORMANCE": 16.380322579909503,
"PIC_STANDARD_STORAGE": 24.570483872279066
}
}Storing it in this two-dimensional dict has the advantage that we can work on it in various ways later. Of course, the simplest of all operations would be to calculate the total cost for the compartment by passing the bill we just got from the API to the following function:
def CompartmentCost(bill):
# adds up all the cost items in the given
# compartment bill and returns it as a number
total=0
for service in bill:
for resource in bill[service]:
total+=bill[service][resource]
return totalWe can do the very same thing for the tenancy as a whole, by just using the API call without the tagging option. It will return the same data structure, so we can also use the CompartmentCost function to add up cost. This will come in handy later on.
Cost for multiple Compartments and Subcompartments
The function above returns the cost for a single compartment. As we know from the previous blog post , the API, while supporting a query for more than one compartment, only provides total cost per query. So to retain information for cost by compartment for all compartments, or for all subcompartments, we will need to call the API once for each compartment we’re interested in. For this to work, we need two things: A list of all compartments and a way to relate them to each other. The metering API does not provide either, so for this we need to go back to OCI. Fortunately, there is a Python SDK for OCI, so getting a list of compartments and their relationship is relatively straight forward.
When asked for all compartments, the OCI API returns a list of records like this one:
{
"compartment-id": "<ocid_tenancy>",
"defined-tags": {},
"description": "some freetext description",
"freeform-tags": {},
"id": "<ocid_compartment>",
"inactive-status": null,
"is-accessible": null,
"lifecycle-state": "ACTIVE",
"name": "Compartment Name",
"time-created": "2018-05-20T17:48:13.636000+00:00"
}The three fields of interest to us are:
- id – The OCID of the compartment. Since this, unlike the compartment name, is unique, we can safely use it to query for the compartment in the metering API.
- compartment-id – This is the OCID of the parent of this compartment. We can use this to relate one compartment to another. This field is null for the root compartment, which is the tenancy itself.
- lifecycle-state – Usually, we only want to see information for compartments that have not been deleted. Since deleted compartments remain visible in the system for at least 90 days, filtering out these compartments is usually a good idea.
Using this data about compartments, we can now query the metering API for a set of compartments.
Traversing the Compartment Tree
In many cases, it will be interesting to see the cost of compartments, including all their subcompartments. As we’ve seen above, the OCI API delivers a list of compartments with pointers to their parent compartments. This is sufficient to use recursion to return a tree view of all compartments.
Using the Python SDK, we can easily get a list of all compartments:
compartmentlist=oci.pagination.list_call_get_all_results(
login.list_compartments,tenancy,access_level="ANY",
compartment_id_in_subtree=True)(For a detailed description of the parameters for this call, see the documentation. For a full example, see below.)
The root compartment will be missing in this list. To add it, we do something like:
compartmentlist.data.append(login.get_compartment(tenancy).data)Now compartmentlist.data will be a list of compartment records as described above.
To print any part of this tree, we can use a recursive function like this:
def PrintCompartmentTree(compartment,indent,compartmentlist):
thiscompartment=[comp for comp in compartmentlist if comp.id==compartment][0]
# will always return only a single element because compartmentID is unique
if (thiscompartment.lifecycle_state != 'DELETED'):
print(f'{indent}{thiscompartment.name}')
for c in compartmentlist:
if c.compartment_id == compartment: # if I'm the parent of someone
PrintCompartmentTree(c.id,indent+thiscompartment.name+'/',compartmentlist)
returnTo call it for a specific compartmentID as the root of the tree, we’d call
PrintCompartmentTree(compartmentID,"",allcompartments)Next, let’s see how we can use this scheme to gather the cost of a compartment and all its subcompartments.
These are the basic steps:
- Get the list of all compartments
- Recurse over the compartment tree, starting at the compartment we’re interested in.
- For each of these compartments, get their associated cost
- Add all of it up, reporting on individual subcompartments if required.
Here are the main building blocks for this:
Get the list of all compartments
def ReadAllCompartments(tenancy,login):
compartmentlist=oci.pagination.list_call_get_all_results(login.list_compartments,tenancy,
access_level="ANY",compartment_id_in_subtree=True)
compartmentlist.data.append(login.get_compartment(tenancy).data)
return compartmentlist.data
ociconfig = oci.config.from_file(ociconfigfile,"DEFAULT")
ociidentity = oci.identity.IdentityClient(ociconfig)
allcompartments = ReadAllCompartments(ociconfig['tenancy'],ociidentity)Get the bill for a compartment tree
def GetTreeBills(compartment,compartmentlist,meteringid,start,end,costbyID):
# traverses down all subcompartments of the given compartmentID and
# fetches the bill for each compartment
# expensive because calling the API
# uses the given list as "all compartments"
# uses recursion on this list.
# results are stored in costbyID
thiscompartment=[comp for comp in compartmentlist if comp.id==compartment][0]
if (thiscompartment.lifecycle_state != 'DELETED'):
# get bill for this compartment
costbyID[thiscompartment.id]={}
costbyID[thiscompartment.id]=get_compartment_charges(meteringid,start,end,thiscompartment)
for c in compartmentlist:
if c.compartment_id == compartment: # if I'm the parent of someone
GetTreeBills(c.id,compartmentlist,meteringid,start,end,costbyID)
CompartmentCostbyID = {}
GetTreeBills(compartmentID,allcompartments,meteringdata,startdate,enddate,CompartmentCostbyID)Once this is done, the array CompartmentCostbyID will contain the bills for the compartment with OCID compartmentID and all its subcompartments.
Running Reports
Working on the results, we can get the cost of a (sub)tree of compartments:
def GetTreeCost(compartment,compartmentlist):
# traverses down all subcompartments of the given compartmentID and returns their cost
# uses the given list as "all compartments"
# uses recursion on this list.
# uses the global array "CompartmentCostbyID" as a data source
thiscompartment=[comp for comp in compartmentlist if comp.id==compartment][0]
# will always return only a single element because compartmentID is unique
childrenscost=0
totalcost=0
mycost=0
if (thiscompartment.lifecycle_state != 'DELETED'):
# get cost for this compartment
mycost=CompartmentCost(CompartmentCostbyID[thiscompartment.id])
for c in compartmentlist:
if c.compartment_id == compartment: # if I'm the parent of someone
childrenscost+=GetTreeCost(c.id,compartmentlist)
totalcost=mycost+childrenscost
return totalcostWe can now run various reports on the array CompartmentCostbyID. For example, the following function summarizes the cost per compartment and stores one line per compartment in a table.
def PrintCompartmentTree(compartment,indent,level,maxlevel,compartmentlist,resulttable):
# traverses down all subcompartments of the given
# compartmentID and prints their name and cost
# uses the given list as "all compartments"
# uses recursion on this list.
# uses global "accountbill"
thiscompartment=[comp for comp in compartmentlist if comp.id==compartment][0]
childrenscost=0
totalcost=0
mycost=0
if (thiscompartment.lifecycle_state != 'DELETED'):
# get cost for this compartment
mycost=CompartmentCost(CompartmentCostbyID[thiscompartment.id])
# first get the cost for all the children so we can print the header line
for c in compartmentlist:
if c.compartment_id == compartment: # if I'm the parent of someone
childrenscost+=GetTreeCost(c.id,compartmentlist)
totalcost=mycost+childrenscost
if (level==1):
try: # will fail if we're not reporting on the root compartment
accountcost=CompartmentCost(accountbill)
resulttable.add_row(["Tenancy Total","{:6.2f}".format(accountcost),
"","","",""])
resulttable.add_row(["General Account","{:6.2f}".format(accountcost-totalcost),
"","","",""])
except:
accountcost=0;
if (level <= maxlevel):
resulttable.add_row([indent+thiscompartment.name,"{:6.2f}".format(totalcost),
"{:6.2f}".format(mycost),"{:6.2f}".format(childrenscost),
compartment])
# then recurse into the children to get all their lines printed
for c in compartmentlist:
if c.compartment_id == compartment: # if I'm the parent of someone
PrintCompartmentTree(c.id,indent+thiscompartment.name+'/',level+1,maxlevel,
compartmentlist,resulttable)We can then call this function to get the report data in tabular form and print the resulting table in various formats.
outputtable=PrettyTable(["Compartment","Total","Local","Subcompartments","OCID"])
PrintCompartmentTree(compartmentID,"",1,maxlevel,allcompartments,outputtable)
# ASCII output
print(outputtable.get_string(title="Compartment Cost between "+start_date+" and "+end_date ,
fields=["Compartment","Total","Local","Subcompartments"]
))
# HTML output
print(outputtable.get_html_string(
title="Compartment Cost between "+start_date+" and "+end_date ,
fields=["Compartment","Total","Local","Subcompartments"]
),file=htmlfile)With a little trick, we can also produce CSV:
with open(csvname+".csv","w",newline='') as csvfile:
writer=csv.writer(csvfile)
writer.writerows([outputtable._field_names])
writer.writerows(outputtable._rows)Sample ASCII output would look like the table below. In this example, maxlevel was set to 2, so that only the first two levels of subcompartments are actually reported, although of course all levels are taken into account. This is nice for a “manager level” report – and exactly what we wanted for the example use case mentioned in the beginning.
+------------------------------------+----------+---------+-----------------+
| Compartment | Total | Local | Subcompartments |
+------------------------------------+----------+---------+-----------------+
| Tenancy Total | 29610.82 | | |
| General Account | 6015.32 | | |
| demo | 23595.50 | 1211.97 | 22383.52 |
| demo/Dambusters | 69.70 | 0.00 | 69.70 |
| demo/DevCSCompartment | 0.00 | 0.00 | 0.00 |
| demo/Easyrider | 10291.60 | 462.46 | 9829.14 |
| demo/Learning | 575.74 | 575.74 | 0.00 |
| demo/ManagedCompartmentForPaaS | 754.12 | 754.12 | 0.00 |
| demo/Pegasus | 5492.17 | 760.21 | 4731.96 |
| demo/Wizards | 5200.19 | 3865.81 | 1334.38 |
+------------------------------------+----------+---------+-----------------+One thing you will notice in the above example is the line General Account. It reports on all the cost reported by the API that is not associated with any compartment. As described in the blog post about the API, cost for some Classic services are reported like this. In the function above, the total cost for the account is stored in the global array accountbill. The cost of the root compartment and all its subcompartments are subtracted from the cost for the whole account. The difference is reported as General Account cost – the cost for all Classic services. To get a more detailed view on these costs, we can use a slightly different function to report on services and resources by compartment. This function, again, contains code to deduct any cost associated to a compartment from the total account bill, which allows the reporting of Classic services:
def PrintDetailTree(compartment,indent,compartmentlist,resulttable):
# traverses down all subcompartments of the given compartmentID
# and prints their name and cost details
# uses the given list as "all compartments"
# uses recursion on this list.
global unaccounted # tracking how much we accounted for in compartments
# modifying this global variable here !!
thiscompartment=[comp for comp in compartmentlist if comp.id==compartment][0]
# will always return only a single element because compartmentID is unique
if (thiscompartment.lifecycle_state != 'DELETED'):
# get bill for this compartment
mybill=CompartmentCostbyID[thiscompartment.id]
for service in mybill:
for resource in mybill[service]:
try:
# deduct what we have here from the unaccounted stuff
unaccounted[service][resource]-=mybill[service][resource]
except NameError:
pass
if (mybill[service][resource]>=0.005): #only print what's not rounded to 0.00
resulttable.add_row([indent+thiscompartment.name,
"{:6.2f}".format(mybill[service][resource]),
service,resource
])
# then recurse into the children to get all their lines printed
for c in compartmentlist:
if c.compartment_id == compartment: # if I'm the parent of someone
PrintDetailTree(c.id,indent+thiscompartment.name+'/',compartmentlist,resulttable)
if (indent==''): # report unaccounted
try:
for service in unaccounted:
for resource in unaccounted[service]:
if (unaccounted[service][resource]>=0.005):
#only print what's not rounded to 0.00
resulttable.add_row(["General Account",
"{:6.2f}".format(unaccounted[service][resource]),
service,resource
])
except NameError:
pass
returnThe resulting table will include a section showing the remaining, Classic cost:
+-------------------+--------+-------------------+-----------------------------------------------------------------+
| General Account | 217.74 | AUTOANALYTICS | ANALYTICS_EE_PAAS_OM_OCI |
| General Account | 40.50 | AUTOBLOCKCHAIN | OBCS_EE_PAAS_ANY_TRANSACTIONS_HOUR |
| General Account | 383.07 | CASB | CASB_IAAS_ACTIVE_ACCOUNTS_COUNT |
| General Account | 3.72 | IDCS | Enterprise-STANDARD |
| General Account | 0.15 | IDCS | Oracle Identity Cloud Service - Consumer User - User Per Month |
| General Account | 24.68 | INTEGRATIONCAUTO | OIC-A E BYOL |
| General Account | 0.87 | JAAS | JCS_EE_PAAS_ANY_OCPU_HOUR_BYOL |
| General Account | 0.58 | JAAS | JCS_EE_PAAS_GP_OCPU_HOUR |
| General Account | 241.15 | MobileStandard | MOBILESTANDARD_NUMBER_OF_REQUESTS |
| General Account | 123.49 | OMCEXTERNAL | ENTERPRISE_ENTITIES |
| General Account | 32.05 | OMCEXTERNAL | LOG_DATA |
| General Account | 37.80 | OMCEXTERNAL | SECURITY_ENTITIES |
| General Account | 120.97 | OMCEXTERNAL | SECURITY_LOG_DATA |
| General Account | 115.92 | VISUALBUILDERAUTO | VBCS-A OCPU |
+-------------------+--------+-------------------+-----------------------------------------------------------------+Summary
The above code segments demonstrate how to use the Oracle Cloud Metering API with Python and combine it with the OCI Python SDK to generate custom reports on cloud cost. Specifically, they demonstrate how to use compartment relationships gathered from the OCI Python SKD to enhance the cost information from the Metering API with a compartment view. They also show how to easily create different reports with the collected data. Using these samples, you should be able to quickly develop your own, custom reports.
References
- Blog about the Metering API
https://www.ateam-oracle.com/a-first-look-at-the-oracle-cloud-metering-api - An example for a python IDE with debugger
https://wingware.com/ - Documentation for the OCI Python SDK
https://oracle-cloud-infrastructure-python-sdk.readthedocs.io/en/latest/api/landing.html
Stefan Hinker
After around 20 years working on SPARC and Solaris, I am now a member of A-Team, focusing on infrastructure on Oracle Cloud.
Previous Post