Skip to main content
Graduate Track

Copying SharePoint Online Document Libraries to Amazon S3 with Amazon AppFlow

Copy SharePoint Online document libraries to Amazon S3 with Amazon AppFlow: the connection, bucket policy, scheduled vs on-demand runs, limits and CLI checks.

KBY Technologies14 min readFoundation12 min
Copying SharePoint Online Document Libraries to Amazon S3 with Amazon AppFlow

Advertising disclosure: this article includes partner links. KBY may earn a commission if you sign up or buy through one, at no extra cost to you. How we make money.

Before you begin

  • A SharePoint Online test site with a document library of non-sensitive test documents.
  • Your Microsoft Entra tenant ID.
  • A lab S3 bucket in the Region where you use AppFlow.

To copy documents out of SharePoint Online into AWS, create an Amazon AppFlow connection to Microsoft SharePoint Online with your Microsoft Entra tenant ID, then create a flow that uses that connection as the source, picks a site and one or more document library folders, and writes to an Amazon S3 bucket in the same AWS Region. Each run puts the documents in a folder named after the run’s execution ID, next to a JSON file of metadata about every document it copied. The key constraint is direction: AppFlow can only read from SharePoint Online, it can only write those documents to S3, and it does not connect to on-premises SharePoint Server.

That makes AppFlow a copy tool, not a sync tool. Nothing you do in the bucket changes SharePoint, and every run writes a new folder rather than updating an earlier one. Whether a run copies everything or only what changed depends on how the flow is triggered, and there are hard limits on folders, files and size per run. This guide explains the moving parts first, then walks through a worked example and a small exercise you can run against a test site and a lab bucket.

#By the end of this guide

  • Explain the path a document takes from a SharePoint document library to an S3 bucket, and which side holds which permission.
  • Choose between the AWS managed client app and your own Entra app registration for the connection.
  • Predict what a scheduled run and an on-demand run will copy.
  • Run a flow once, find its execution ID and read its result from the AWS CLI.
  • Check the copied documents in S3 and undo the lab cleanly.

#What you need before you start

  • A SharePoint Online site with at least one document library that holds a few test documents with nothing sensitive in them.
  • Your Entra tenant ID. In the Microsoft Entra admin center, as at least a Global Reader, browse to Entra ID, Overview, Properties and copy the Tenant ID.
  • An AWS account you are allowed to test in, with an empty S3 bucket for the lab in the Region where you will use AppFlow.
  • An IAM identity that can create AppFlow connections and flows. The AmazonAppFlowFullAccess managed policy includes the UseConnectorProfile permission a flow needs to use a connection; a custom policy has to grant it on the connection’s connectorprofile resource.
  • The AWS CLI, configured for that account and Region, for the read-only checks.

#How a document gets from SharePoint to S3

Three objects do the work, and it helps to keep them apart. A connection (the CLI and API call it a connector profile) stores the details and credentials AppFlow needs to reach SharePoint on your behalf. A flow says what to copy and where to put it: the connection, the SharePoint site, the library folders and the destination bucket. A run is one execution of the flow, and every run gets its own execution ID.

There are two trust boundaries. On the Microsoft side, whoever creates the connection signs in to Microsoft and authorises AppFlow to access the SharePoint data. If you use your own app registration, AWS recommends giving it the Microsoft Graph delegated permissions offline_access, Sites.Read.All and User.Read, which means AppFlow acts on behalf of the person who signed in. Microsoft describes delegated Sites.Read.All as letting the application read documents and list items in all site collections on behalf of the signed-in user. Read is the important word: nothing in that set lets AppFlow change SharePoint.

On the AWS side, the bucket has to let the AppFlow service write into it. When you pick a bucket as a flow’s destination in the AppFlow console, AppFlow attaches the permissions it needs to the bucket policy for you: a statement for the service principal appflow.amazonaws.com allowing s3:PutObject, s3:AbortMultipartUpload, s3:ListMultipartUploadParts, s3:ListBucketMultipartUploads, s3:GetBucketAcl and s3:PutObjectAcl on the bucket and its contents. That is a change to the bucket, so you back the policy up before you start. The bucket must be in the same Region as the flow, and AppFlow does not support buckets in another AWS account.

Rendering diagram...

The diagram shows the one-way path. The connection holds the Microsoft authorisation, the flow narrows it to a site and specific folders, and each run lands in S3 under its own execution ID. If either boundary is wrong, the run is what tells you.

Close-up of server racks in a data center highlighting modern technology infrastructure.
Photo by panumas nikhomkhai on Pexels

#Two ways to authorise the connection

By default the Use AWS managed client app checkbox is on. With it, the only thing you take from Microsoft is the tenant ID: you enter it in the Custom authorization tokens URL and Custom authorization code URL fields, choose Continue, sign in to Microsoft when prompted and choose Authorize. No client ID or secret is involved.

The alternative is your own app registration in Entra. It is more work, but you control the credentials: you can rotate them, revoke them and decide who manages them. The registration needs a redirect URL in the form https://region.console.aws.amazon.com/appflow/oauth, with your Region code in place of region (for US East (N. Virginia), https://us-east-1.console.aws.amazon.com/appflow/oauth), the delegated permissions above and a client secret. You then clear the checkbox and enter the client ID and client secret. For a lab, the managed app is fine. For production, ask who owns app registrations in your tenant before you choose; it is their decision as much as yours.

One practical snag: the sign-in is a three-legged OAuth exchange, and AWS notes that if multi-session support is turned on in the AWS Management Console, you need to turn it off for the connection set-up to complete.

#What each run copies

The trigger decides the scope of every run, and this is where most surprises come from.

  • On demand. Every run is a full transfer: AppFlow copies every document in the folders you chose, each time. You start it with Run flow on the flow details page, or from the CLI.
  • On a schedule. The first run copies every document in the chosen folders. After that, each run copies only files created or changed in SharePoint since the previous run. If a run is still going when the next one is due, AppFlow skips the next one.

Either way, each run writes into its own execution-ID folder. A month of daily on-demand runs is a month of full copies, not one up-to-date copy.

The limits are worth knowing before you design anything. A flow can transfer at most 17 document library folders. A single file can be at most 250 GB. A run can transfer at most 10,000 files and at most 250 GB in total. The usual S3 extras are not available for this source either: you cannot catalogue the output in the AWS Glue Data Catalog, because documents are unstructured data, and AppFlow’s partitioning options do not apply, because all SharePoint output is partitioned only into execution-ID folders.

#In practice: a policy library copied for an audit

Suppose a compliance team keeps signed policies in a SharePoint site called Policies, in a library with a folder for each year, and an auditor wants a copy of the current year’s folder in an S3 bucket the audit team already uses.

You create the connection, then a flow called policies-audit-copy. Under Source details you choose Microsoft SharePoint Online, the connection, API version v1.0, the Policies site, and tick the folder for this year under Selected resources. Ticking a folder includes everything in its subfolders too. Under Destination details you choose Amazon S3, the audit bucket and a prefix of sharepoint-policies, which becomes a folder in the bucket. Because the auditor wants one copy, not a feed, the trigger is Run on demand.

After one run, the bucket holds a JSON metadata file named after the execution ID and a folder with the same name that contains the copied folders and documents. The metadata includes fields such as Name, Size, Web Url, Created DateTime and Last Modified DateTime, and AppFlow writes only the fields that apply to each document type. That file is what you hand over with the copy: it describes each document the run transferred.

The decision that matters is the trigger. Had you picked a daily schedule, the second run would contain only documents changed since the first, in a new execution-ID folder, and the auditor would have to combine folders to see the full set.

#Practice run: copy one test folder to a lab bucket

The objective is to copy one SharePoint test folder to S3 with an on-demand flow, prove the copy from the CLI, and then put everything back. Use a test site and a lab bucket only; nothing here should touch a library with real documents in it.

  1. Back up the lab bucket’s policy before AppFlow edits it:
    1aws s3api get-bucket-policy --bucket amzn-s3-demo-bucket --query Policy --output text > policy-before-appflow.json

    Keep the file. If the lab bucket is new and has no policy yet, there is nothing to save; write that down, because the restore is then to remove the policy rather than re-apply a file.

  2. In the AppFlow console, choose Connections, choose Microsoft SharePoint Online under Connectors, then Create connection. Enter the tenant ID in both URL fields, leave Use AWS managed client app on, name the connection lab-sharepoint, choose Continue and then Authorize. The connection should appear in the Connections table.
  3. Choose Create flow and name it lab-sharepoint-to-s3. Flow names allow letters, numbers and the characters !@#.-_, but no spaces.
  4. Under Source details, choose Microsoft SharePoint Online, lab-sharepoint, API version v1.0, your test site, and tick one test folder.
  5. Under Destination details, choose Amazon S3, the lab bucket and the prefix lab-run. Set the trigger to Run on demand, work through the remaining pages and choose Create flow.
  6. Start the run from the CLI:
    1aws appflow start-flow --flow-name lab-sharepoint-to-s3

    For an on-demand flow, the response includes a flowStatus and the run’s executionId. Note the ID.

  7. Read the result:
    1aws appflow describe-flow-execution-records --flow-name lab-sharepoint-to-s3

    Find the entry with your execution ID. Its executionStatus moves from InProgress to Successful or Error, and executionResult shows bytesWritten, recordsProcessed and, on failure, errorInfo.executionMessage.

  8. List what landed:
    1aws s3 ls s3://amzn-s3-demo-bucket/lab-run/ --recursive --human-readable --summarize

The pass condition is a Successful status, a metadata JSON file and a folder both named with your execution ID, and a summary object count that makes sense for the folder you ticked: your documents plus the metadata file. Stop if the status is Error, if the listing is empty, or if you see documents from a folder you did not tick, and read the error message before changing anything.

To undo the lab, delete the flow on the Flows page (the console asks you to type delete), then delete lab-sharepoint on the Connections page. Check the listing shows only your execution-ID objects under lab-run/, then remove that prefix with aws s3 rm s3://amzn-s3-demo-bucket/lab-run/ --recursive. Finally restore the bucket policy with aws s3api put-bucket-policy --bucket amzn-s3-demo-bucket --policy file://policy-before-appflow.json, or, if the bucket had no policy before, remove it with aws s3api delete-bucket-policy --bucket amzn-s3-demo-bucket. If you consented to an app in Entra for the lab, ask a Cloud Application Administrator to review it under Entra ID, Enterprise apps, All applications, Permissions; admin consent can be revoked there, but user consent cannot be revoked in the portal.

Close-up of a person navigating online checkout on a laptop screen in a cozy indoor setting.
Photo by Pavel Danilyuk on Pexels

#How to check the outcome of a run

A green status is not proof on its own. Check three things, in this order, and each one tells you something different.

  1. The run record. describe-flow-execution-records shows the status and how much was written. A Successful run with bytesWritten of zero suggests there was nothing to copy, which for a scheduled flow can mean nothing changed since the last run.
  2. The objects. The aws s3 ls summary gives an object count and total size for the execution-ID folder. Compare the count with the number of documents in the SharePoint folder, plus one for the metadata file.
  3. The metadata file. Open it and spot-check a few entries: the Name, Size and Last Modified DateTime should match what SharePoint shows for the same documents.

If the counts disagree, look at the quotas before you look for a fault: a folder with more than 10,000 files or more than 250 GB will not fit in one run.

#Pitfalls that stop or skew a transfer

  • The bucket is in another Region or another account. AppFlow needs the bucket in the same Region as the flow and does not support cross-account buckets. Create the bucket where the flow runs.
  • Expecting a mirror. On-demand runs are full copies each time and scheduled runs after the first are only the changes, each in a new execution-ID folder, so no single folder is a live copy of the library.
  • Ticking too many folders. The limit is 17 folders per flow, so pick the parent folder when you want everything under it.
  • A connection that no longer works. AWS notes that an expired user login can return a status code 400 error; the documented fix is to create a new connection and delete the old one, or use a connection with valid credentials.
  • A custom IAM policy without UseConnectorProfile. Flow creation fails because the identity cannot use the connection. Add the permission for that connectorprofile resource.
  • Reading the copies later. The bucket policy AppFlow adds lets AppFlow write; it does not let your applications read. If an EC2 instance later gets AccessDenied reading the documents, that is a separate permission problem, covered in fixing S3 AccessDenied when an EC2 instance role reads a bucket.

Cancelling a run does not undo it. AWS states that data a cancelled run has already written stays in the destination, and you are still charged for what it processed.

#Recap

  • AppFlow copies SharePoint Online documents one way, into S3 only, and never connects to SharePoint Server.
  • The connection holds the Microsoft authorisation; the flow chooses the site, folders and bucket; each run lands under its own execution ID with a metadata JSON file.
  • On demand means a full copy every run; a schedule means everything once, then changes only.
  • Back up the bucket policy first, because the console edits it for you.
  • Prove a run with the execution record, the S3 listing and the metadata file together.

#Before you point a flow at a real library

Outside a lab, the copy is data leaving SharePoint’s own controls, so the questions change. Agree with the site owner which folders may be copied and who may read the bucket. Decide who authorises the connection, because AppFlow’s access to SharePoint comes from the Microsoft sign-in that person approves. If your organisation wants control over the credentials, use your own app registration rather than the managed app, and have whoever owns Entra applications review its permissions. If you want encryption with your own key, choose Customize encryption settings (advanced) on the connection and pick a customer managed AWS KMS key; otherwise AppFlow uses a key that AWS manages for you.

Then run the flow once on demand, check it the way the lab did, and only switch to a schedule when you have seen a full copy that matches. Record the bucket policy before and after, so anyone can see what AppFlow added. If you later need to know who changed something in the account, the approach in using CloudTrail Event History to find who changed a security group is a good place to start. Escalate rather than widen permissions if a run fails with an authorisation error you cannot explain from the connection or the bucket policy.

About this page

Credited to

Dates

Published

Sources

This page lists no external sources.

Corrections

No corrections are listed for this page. Some review sweeps corrected pages without naming each one; those are in the corrections log.

Reader Interaction

Comments

Add a thoughtful note on Copying SharePoint Online Document Libraries to Amazon S3 with Amazon AppFlow. Comments are checked for spam and held for moderation before appearing.

Loading comments...

Discover more

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.

Build practical engineering skills.

Receive new lessons, learning paths, practical exercises and early-career guidance.