Custom data API requirements should explain what data you need, where it comes from, how it should behave and how your systems will receive it. Before evaluating data extraction and API solutions, define those requirements in terms your business and engineering teams can both test.
“Product data updated regularly” is not enough to establish a reliable project scope. It leaves questions about source coverage, field definitions, historical availability and what happens when a source stops responding.
A useful brief separates the data requirements from the technical interface. It also establishes the evidence needed to approve a sample, accept delivery and manage future changes.
Quick Answer
To scope a custom data API, document the business use case, approved sources, record definitions, required fields, transformation rules, collection volume, request traffic, historical depth and freshness targets. Then specify delivery, security, error handling and acceptance tests. Separate verified capabilities from assumptions that still require sample validation.
Define the Data You Need
Start With the Business Decision
Begin with the decision or workflow the data will support.
A pricing team may need comparable competitor offers before its morning review. A product application may need low-latency lookups. A research team may prioritise historical coverage over frequent updates.
Those requirements should not produce identical APIs.
Write a short purpose statement, such as:
Provide the pricing team with comparable product offers from approved retailers, refreshed before each scheduled pricing review.
Then identify the consumer, the action it will take and the consequence of incomplete or stale information.
This creates a basis for prioritisation. A field that does not support the first release can remain optional rather than expanding the initial scope.
Separate Requirements From Interface Design
An API requirements document describes the business need, dataset, service expectations and acceptance conditions.
A custom API specification describes how software interacts with the service, including endpoints, parameters, schemas, authentication and responses. OpenAPI provides a standard way to describe these HTTP API interfaces.
You need both perspectives.
| Document component | Questions it answers |
|---|---|
| Business requirements | Who needs the data, and what will they do with it? |
| Data requirements | Which sources, records, fields and historical periods are included? |
| Interface specification | How do clients request and receive data? |
| Service requirements | How fresh, available and responsive must the service be? |
| Acceptance plan | What evidence demonstrates that delivery meets the agreement? |
When evaluating a custom data API, request a sample response alongside the broader scope. A technically valid response does not, by itself, demonstrate sufficient source coverage or useful data.
Build an Approved Source Inventory
List sources individually rather than requesting “all major websites.”
For each source, record the relevant domain, dataset or endpoint, geographic coverage, language and access method. Include representative pages or example records so everyone is evaluating the same information.
Your source inventory should identify:
- The source and business reason for including it.
- The countries, categories or locations in scope.
- Whether access uses an official API, licensed feed or permitted website collection.
- Any authentication, subscription or licensing requirements.
- The fields expected from that source.
- Known access or coverage limitations.
- Who approves its inclusion.
A data source API and a website may expose different information. Validate the required fields against the access method you intend to use.
If web data extraction is part of the project, assess feasibility and permitted use before promising coverage. Public visibility does not remove privacy obligations: regulators have explicitly stated that publicly accessible personal information remains subject to data protection laws in most jurisdictions.
Have the appropriate reviewers assess terms, licensing, privacy and downstream use restrictions.
Define What One Record Represents
Agree on the unit of data before estimating volume.
“One product” could mean a parent product, a size or color variant, a seller offer or an observation of that offer at a particular time.
Those are different records.
For a pricing project, a useful definition might be:
One observation represents one seller’s offer for one product variant, in one market, at one collection time.
Then distinguish the stable entity identifier from the observation identifier. The offer may remain the same while its price changes across many observations.
This decision affects deduplication, history, matching and billing. Make it explicit before comparing provider quotes.
Create a Field Dictionary
A field list should define meaning as well as spelling.
For each field, document its type, whether it must be present, whether it can be null, its source and the validation rule. OpenAPI schemas can describe input and output types, but the business meaning still needs clear documentation.
An illustrative offer-observation dictionary could include:
| Field | Type | Requirement | Meaning |
|---|---|---|---|
| observation_id | String | Required, non-null | Unique identifier for this observation |
| offer_id | String | Required, non-null | Stable identifier for the source-specific offer |
| product_id | String | Nullable | Matched internal product identifier, when available |
| price_amount | Decimal string | Nullable with a reason | Advertised item price under the agreed pricing rules |
| currency | String | Required when price exists | Currency associated with the price |
| availability | Enumerated string | Required | Agreed status, including an explicit unknown value |
| source_url | String | Required | Location supporting the observation |
| observed_at | Timestamp | Required | Time the information was collected |
| source_updated_at | Timestamp | Nullable | Source-reported update time, when available |
A required field and a non-null field are not necessarily the same thing. An unavailable price may need to appear as null with an explanation rather than disappear from the response.
Also define units, timestamp conventions, rounding and monetary precision. Do not leave those decisions to individual integration developers.
Make Business Rules Explicit
Business rules determine whether records are comparable and usable.
For price monitoring, specify whether a price includes tax, shipping, membership conditions or coupons.
A standard advertised price should not be silently mixed with a price available only to selected customers.
Document rules for:
- Product and entity matching.
- Variant and seller separation.
- Currency and unit normalization.
- Duplicate detection.
- Missing, invalid and conflicting values.
- Outlier handling.
- Source precedence.
- Removed or unavailable records.
- Corrections to previously delivered data.
For example, define whether an unavailable product remains in the dataset with a changed status or disappears from the current view.
Do not silently substitute yesterday’s value when today’s collection fails. If previous values are served, identify their observation time and stale status.
Verify Historical Depth
Separate historical backfill from future retention.
Backfill concerns information that already exists. Retention concerns how long newly collected records will remain available.
Request the earliest available date, source-by-source coverage and historical granularity. “Two years of history” might mean daily snapshots, monthly summaries or irregular observations.
A current product page cannot establish every price that appeared on it last year. Historical claims need evidence from retained observations, archives or another suitable source.
When assessing ecommerce data feeds, validate history separately for prices, reviews and other entities. A review’s publication date is not proof that historical product prices are available for the same period.
Ask for a backfill sample containing dates, gaps and provenance before including historical coverage in the final agreement.
Specify Delivery and Acceptance
Separate Collection From Request Volume
Collection workload and customer API traffic are different quantities.
A provider might collect data four times daily while your application queries the resulting dataset thousands of times between collection runs.
Distinguish these volume measures:
| Volume measure | What to specify |
|---|---|
| Unique entities | Products, listings, locations or other tracked objects |
| Observations | Records generated through repeated collection |
| Source requests | Calls or page fetches required for acquisition |
| Client API requests | Calls made by your applications |
| Peak traffic | Requests per second and concurrent requests |
| Payload volume | Typical and maximum response sizes |
| Historical backfill | Initial records or bytes to deliver |
| Growth | Expected increases in sources, entities and usage |
For an illustrative project with 10,000 offers collected four times daily, the target is 40,000 offer observations per day.
That does not mean 40,000 source requests or 40,000 client API calls. Source pagination, batching, retries and consumer behavior affect those separately.
Agree on rate limits, burst handling and usage quotas. API gateways commonly use quotas to control request rates and protect shared service capacity.
Define Freshness, Not Just Frequency
“Hourly updates” describes a schedule. It does not fully describe how old the delivered data may be.
Separate the source’s own publication timing from collection, processing and delivery. Also distinguish data freshness from API response latency.
A fast API can return old information.
Useful timestamp definitions include:
- source_updated_at: when the source says the information changed.
- observed_at: when the collector observed it.
- available_at: when the processed record became accessible.
If the source does not expose its update time, do not claim to measure exact source-change-to-delivery latency.
Define a measurable freshness objective instead, such as a proposed requirement that 99% of tracked offers have a successful observation no more than seven hours old. Specify the evaluation interval and how missing observations count.
Google’s data-quality guidance treats freshness and completeness as separate dimensions, which is useful when designing these checks.
Choose the Delivery Pattern
Select delivery according to the consumer’s workflow, not simply because an API endpoint is available.
| Delivery pattern | Suitable use | Requirements to define |
|---|---|---|
| Synchronous API | Application lookups and filtered queries | Latency, pagination, filters and rate limits |
| Asynchronous export | Large extracts and backfills | Job status, completion notice and download expiry |
| Webhook | Notifications about new or changed data | Authentication, retries, replay and duplicate handling |
| Scheduled files | Recurring analytics ingestion | Format, destination, partitioning and delivery deadline |
| Warehouse delivery | Data consumed directly in analytics systems | Schema mapping, permissions and load verification |
For long-running operations, Microsoft recommends an asynchronous pattern that acknowledges the request and exposes a status endpoint rather than keeping the client waiting for completion.
A managed data delivery service may offer multiple delivery routes. Confirm which route is included and who owns ingestion failures at the destination.
For webhooks, specify whether events can repeat or arrive out of order. Stripe’s documentation illustrates why event identifiers, duplicate handling and ordering assumptions must be explicit.
Document the Interface Contract
Translate the approved data requirements into an interface engineers can implement.
Include endpoints, authentication, request parameters, response schemas and examples. Document filters, page limits and sorting behavior. Pagination and filtering help clients retrieve the required subset without unnecessarily large responses.
Also specify:
- How clients retrieve changes since a previous synchronization.
- How deletions or removals are represented.
- Whether a paginated export reflects a consistent snapshot.
- Error codes and machine-readable error details.
- Retryable versus non-retryable failures.
- Versioning and deprecation policies.
- Test credentials and a sandbox or sample endpoint.
Define what happens when a source is unavailable but the delivery API remains online. Returning a successful HTTP response with stale data should not conceal a collection failure.
Set Security and Usage Boundaries
Document authentication and authorization separately.
Authentication establishes the calling client’s identity. Authorization determines which datasets, fields and operations that client may access.
OpenAPI can describe security schemes such as API keys and OAuth2, but implementation and enforcement still need testing.
Include credential rotation, transport encryption, tenant isolation where relevant, logging, retention and incident contacts in the security review.
Also document permitted data uses. Internal analysis, customer-facing display, redistribution and model training may require different approvals or licensing terms.
Collect only the information needed for the agreed purpose, and assign responsibility for privacy reviews and deletion requirements where personal data is involved.
Agree on Measurable Acceptance Tests
Avoid approving a project against a vague promise of “high-quality data.”
Define each measure, denominator, test population and observation period. Include source failures rather than measuring quality only among records that were successfully returned.
| Acceptance area | Example test |
|---|---|
| Coverage | Compare delivered entities with an agreed reference list |
| Completeness | Measure required field population across eligible records |
| Accuracy | Compare a representative sample with source evidence captured at collection time |
| Uniqueness | Check the agreed record keys for duplicates |
| Freshness | Measure observation age across the full tracked population |
| Performance | Test percentile latency under defined load and payload conditions |
| History | Verify dates, gaps and provenance in the backfill |
| Recovery | Simulate failed delivery, retries and replay |
| Compatibility | Validate responses against the approved schema |
Data-quality monitoring should examine contents as well as whether the pipeline ran successfully. Google’s guidance distinguishes basic pipeline observability from checks that assess the data itself.
Separate API availability, collection success, freshness and support response times. One uptime percentage cannot describe all four.
Use an Illustrative Scope Summary
The following example shows how requirements can become a compact project brief. These values are proposed planning assumptions, not standard benchmarks or provider commitments.
| Scope item | Illustrative requirement |
|---|---|
| Business purpose | Support scheduled competitor pricing reviews |
| Sources | Three approved retail sources in one market |
| Record unit | One seller offer for one variant at one observation time |
| Tracked volume | 10,000 offers |
| Collection frequency | Four scheduled runs daily |
| Observation target | 40,000 daily observations before accounting for collection failures |
| Consumer traffic | Up to 100,000 API requests daily, with a defined peak of 20 requests per second |
| History | A requested 90-day backfill, subject to verified availability |
| Retention | Twelve months of observations collected after launch |
| Delivery | JSON API for queries and a daily file for reconciliation |
| Freshness | An agreed observation-age target measured across tracked offers |
Attach the source inventory, field dictionary, transformation rules and acceptance plan to this summary.
Where feasibility remains uncertain, mark the requirement as pending validation. Do not turn a desired capability into a contractual assumption by leaving its status unclear.
Control Cost and Scope Changes
Ask for pricing units that match the workload.
Clarify whether charges apply to tracked entities, observations, API calls, delivered records, storage, backfills or a fixed managed feed.
Include setup, source maintenance, failed attempts, retries, egress and overages in the discussion. Confirm which items are included rather than assuming they are free.
Assign owners for approving new sources, fields, markets and refresh schedules. Require a cost and delivery assessment before changes enter production.
A source redesign may be routine maintenance. Adding a new market may be a scope expansion. Establish that distinction before either occurs.
Contact Us
Prepare a source list, sample records, required fields, expected volume and freshness targets before requesting an estimate. Include your historical requirements and preferred delivery destination.
Use the Contact Us page to discuss custom data API requirements with TagX and identify which assumptions should be validated through a sample dataset or test endpoint.