Home > Cloud Cruiser 4 > Collecting, transforming, and publishing > Amazon Web Services

Amazon Web Services

Cloud Cruiser’s Amazon Web Services (AWS) solution retrieves consumption data using the AWS Detailed Billing Report API. The template workbook provided for it uses passthrough rates for AWS services, passing along the rates and charges set by AWS, but you have the ability to set your own rates instead.

For more detailed information about collecting Amazon Web Services data, see the following topics:

Data collection overview

Amazon’s Programmatic Billing Access provides hourly detailed billing reports in CSV format, zipping the CSV files and storing them in an S3 (Simple Storage Service) bucket. The AWS Collector downloads these files to <working_dir>/usage_files/aws_detailed_reports and converts them to Cloud Cruiser record files. These files are named aws-detailed-<accountId>-<year>-<month>_<utcTimestamp>.csv.zip, for example aws-detailed-719988776655-2014-9_20140903T200458.csv.zip.

AWS generates a detailed billing report every hour. For complete information about the contents of a detailed billing report, see Understand Your Usage with Detailed Billing Reports in the AWS documentation. For information about the information Cloud Cruiser collects from AWS, see Dimensions and measures collected from AWS.

Because AWS can unpredictably add or change the services it bills for, there is no way for you to create a static set of services in Cloud Cruiser that would remain able to match those services and charge your customers for all AWS usage. To avoid collecting measures that go uncharged, you must create new services from your collected AWS data before publishing that data. The template workbook for AWS includes a flow that does this.

CloudSmart-Now prerequisites

If you want to use the CloudSmart-Now solution for AWS, you must complete the following tasks:

Before you begin

Before creating an Amazon Web Services collection, you must set up your AWS account to work with the Cloud Cruiser collector.

To set up your AWS account

  1. In AWS, turn on detailed billing reports for the account for which you want to collect data.
    For instructions, see Understand Your Usage with Detailed Billing Reports in the AWS documentation. If you are using consolidated billing, perform this step for the payer account, not the linked accounts. For more information about consolidated billing, see Pay Bills for Multiple Accounts with Consolidated Billing in the AWS documentation.
    • Note the name of the Amazon S3 bucket you specify.
    • In the list of reports to receive, select both Monthly report and Detailed billing report with services and tags.
  2. Using Amazon's Identity and Access Management (IAM) service, ​create an AWS user with an access key ID, a secret key, and secure access to the specified S3 bucket​.
    You could simply grant this access to the account owner, but Cloud Cruiser recommends creating a user specifically for billing access​. If you are using consolidated billing, create a user for the payer account.

Configuring Cloud Cruiser to collect, transform, and publish AWS data

The CloudSmart-Now configuration for AWS is configured to work with minimal configuration. If the default configuration does not suit your needs, see the information in Advanced AWS collection options to guide your decisions and changes.

To collect, transform, and publish AWS data

  1. Modify the Amazon Web Services data source named AWS1 to work with your AWS account:
    1. Navigate to Data > Setup > Sources.
    2. Click the AWS1 data source.
    3. In the Access Key Id field, enter the access key ID for an AWS user who has access to billing information.
    4. In the Secret Key field, enter the secret key for this user.
    5. Click Save.
  2. Create an AWS workbook using the Amazon Web Services template.
    1. Navigate to Data > Workbooks.
    2. In the list of workbooks, right-click and select Add New Workbook.
    3. Enter a Workbook Name and (optionally) a Workbook Description.
    4. Check the Create Workbook from Template box, and then select the Amazon Web Services template.
    5. Click OK.
      A new workbook opens, already containing the necessary collections and workflow.
    6. Click Save to save the workbook.
  3. (Optional) If you are not using AWS tags to track resource owners and departments, modify the workbook to use your AWS Resource Ownership data source. For more information about  tracking resource ownership, see Assigning resource ownership.
    1. In the Workbook ribbon, click Paramaters.
    2. In the Workbook Parameters section, change the value of the UseOwnershipFile parameter to true.
    3. Click Save and Close in the dialog box.
    4. Click Save to save the workbook.
  4. Test your workbook to ensure it collects, transforms, and publishes your data correctly.
    1. In the Workbook ribbon, click Run.
      The Run Workbook dialog box opens.
    2. In the Select Date field, enter the current date.
    3. Select all of the collections and flows, and then click Run.
    4. After the run finishes, click View Log at the bottom left of the dialog box to open the Monitoring window, where you can view the log for any specific collection or flow.
    5. If your workbook test run did not complete successfully, or if it did not provide the data you need, edit the workbook and then run another test.
  5. When you are done editing your workbook delete any data loads from your test runs.
    1. Navigate to Data > Processing > Published Loads.
    2. Select one or more entry rows and click Delete Load(s) at the top of the page.
    3. Click Yes to confirm the deletion.
  6. When you are ready to put your workbook into production, start the next procedure, Scheduling regular collection.

Scheduling regular collection

After configuring Cloud Cruiser to collect, transform, and publish your AWS data, create a schedule that defines when and how often to run those processes. You need two schedules for this workbook, one for daily data and one for monthly data.

To schedule regular daily collection

  1. Go to Data > Processing > Scheduling.
  2. In the Schedules section, click New.
    The New Schedule dialog box appears.
  3. In the Job Type field, select Workbook.
  4. In the Name field, enter AWS Daily.
  5. (Optional) In the Description field, enter a description for this schedule, such as Daily schedule for AWS data.
  6. Set Enabled to Yes so that the schedule will run.
  7. In the Frequency section, set the Start Date to the first date that you want the schedule to run, usually today's date.
  8. In the Recurrence field, select Daily.
  9. In the Start Time field enter the time of day the schedule will run, 22:00.
  10. In the Workbook field, select your AWS workbook.
  11. In the Select Date field, select Previous Day.
  12. In the Workbook Components section, select only the AWS1 and ResourceOwnership collections, and the following flows:
    • Ownership
    • ImportServices
    • PublishUsage
    • ImportCustomers
    • HandleUnknowns
    • Cleanup
  13. Click OK.
    The new schedule is added to the list.

To schedule regular monthly collection

  1. Go to Data > Processing > Scheduling.
  2. In the Schedules section, click New.
    The New Schedule dialog box appears.
  3. In the Job Type field, select Workbook.
  4. In the Name field, enter AWS Monthly.
  5. (Optional) In the Description field, enter a description for this schedule, such as Monthly schedule for AWS data.
  6. Set Enabled to Yes so that the schedule will run.
  7. In the Frequency section, set the Start Date to the first date that you want the schedule to run, usually today's date.
  8. In the Recurrence field, select Monthly.
  9. In the Start Time field enter the time of day the schedule will run, 22:45.
  10. In the Workbook field, select your AWS workbook.
  11. In the Select Date field, select Previous Day.
  12. In the Workbook Components section, select only the AWS_EndOfMonth and ResourceOwnership collections, and the following flows:
    • Ownership
    • ImportServices
    • PublishEndOfMonthUsage
    • ImportCustomers
    • HandleUnknowns
  13. Click OK.
    The new schedule is added to the list.
Last modified

Tags

Classifications

This page has no classifications.
© Copyright 2018 Hewlett Packard Enterprise Development LP