Particle API
An API to store & retrieve reusable bits of content as data. Also used to store handlebars templates that can render those bits of content in various ways, and endpoints to do the actual rendering. Also used to store dehydrated article data, and associate them with particles that's been embedded in them.
Authors
- Oliver Edgington oliver.edgington@telegraph.co.uk
- Adriaan Pelzer adriaan.pelzer@telegraph.co.uk
Requirements
⚠️
Something helpful for you developers:
curl -o- -L https://yarnpkg.com/install.sh
yarn global add gulp
Install Xcode command line tools (INSTEAD OF RUBY if you dont have it)
General
Quick Start
Build & Run
Build and start the whole service is simple as: make build && make run
. That command run all related services and mimic the production environment setup on your local machine.
Stop
make stop
stop all service containers that have been started by either make up
or make run
anytime you feel you need a coffee/break.
Start Service standalone
To run the API on your machine, you can run a couple of make targets:
make deps
- Install project dependancies, generally not required if you're using docker. (run whenever you change node modules)make build
- Builds docker image.make run
- Runs the particle API in a docker container.
Links
API URL
http://localhost:5000
Particle CMS URL
requires running the particle CMS, see it's own README for info.
http://localhost:3000
Local S3 Browser URL
requires running the particle CMS, see it's own README for info.
http://localhost:9000
Old CMS URL
requires running the particle CMS, see it's own README for info.
http://localhost:9000
Misc
There are few other useful make targets that you can preview by calling make help
or just make
both in the root directory.
What are Particles?
The term Particle is inspired by the following NYT piece, which, at the time it was published, was perfectly aligned with our thinking around the problem of publication of content for mass consumption.
Particles are components that Articles are made of. It suggests a graph-based approach to publishing, rather than the traditional document-based approach, whereby every Article becomes a series of particles connected by a graph, rather than a single document.
Particles are reusable across multiple Articles, and they have a lifecycle of their own, beyond the context of an Article in which they happen to appear.
We used to call them Embeds.
What is the Particle API?
The Particle API is an interface to the data store where our particles are stored. It allows thin front ends to be built to consume, create, and manage Particles. The Particle CMS is an example of such a front end.
Authentication
The API’s readable endpoints (or any endpoints that doesn’t result in a change being made to the data store) are completely open, and can be queried from anywhere in the world.
The writable endpoints, however, are protected by the following two schemes:
Google user id token
This method of authentication is meant for client where a user is involved, and involves the following steps:
- Implementing Google sign-in for web
- Getting an id_token from Google after the user has signed in (also described at the above link)
- Sending the query variable ?id_token=your-users-id-token with every writable query
Payload signature
This method of authentication is meant for back end clients, where no user is involved, and involves the following procedure:
- Build a string by concatenating the Google Client Secret, Google Client ID, the request path (without query string), the request method in uppercase letters, and the payload. The resulting string will look something like this: cL13NtS3cr3TcL13Nt1D/templatesPOST{"my":"payload","is":"here"}
- Calculate the sha256 digest of this string, using any standard crypto library.
- Sending the query variable ?sig=your-calculated-digest with every writable query
The Google Client ID and Google Client Secret for each environment can be requested from the Editorial Innovation team.
Item store
/lib/itemStore.js The Itemstore is the framework that underlies the particle store, the template store, the article store, and all other future and legacy stores that might be added.
Querying items
- endpoint: GET /itemType-plural
- query variables:
- count: the maximum amount of items to return
- before: timestamp (epoch milliseconds) of earliest item’s last_modified_date (this is an inclusive query)
- after: timestamp (epoch milliseconds) of last item’s last_modified_date (this is an inclusive query)
- key: filter on any exact match of item attribute (top level only, ie shallow)
- return value: List of Items, each with the following structure of auto-generated attributes:
EXAMPLES:
http://particle-api-staging.eip.telegraph.co.uk/itemType-plural (returns 100 particles by default)
http://particle-api-staging.eip.telegraph.co.uk/itemType-plural?count=5 (returns only 5)
http://particle-api-staging.eip.telegraph.co.uk/itemType-plural?count=5&before=1449662938252 (returns the latest 5 particles before and including Wed Dec 09 2015 12:08:58)
http://particle-api-staging.eip.telegraph.co.uk/itemType-plural?count=5&after=1449662938252 (returns the latest 5 particles after and including Wed Dec 09 2015 12:08:58)
http://particle-api-staging.eip.telegraph.co.uk/itemType-plural?key1=value1&key2=value2 (returns all particles where attribute key1 is value1 and key2 is value2)
Querying a single item
- endpoint: GET /itemType-plural/:itemId
- return value: A single Item, with the following structure:
Creating items
- endpoint: POST /itemType-plural
- query variables:
- id_token: (see authentication)
- sig: (see authentication)
- body: Serialised JSON:
NOTE:
author, last_modified_date, publication_date, and id are automatically created - but author can also be supplied with a POST or PUT request. The other automatically create attributes (last_modified_date, publication_date, and id) will be overwritten if supplied
- return value:
Updating items
- endpoint: PUT /itemType-plural/:itemId
- query variables:
- id_token: (see authentication)
- sig: (see authentication)
- body: Serialised JSON:
- return value:
Deleting items
- endpoint: DELETE /itemType-plural/:itemId
- query variables:
- id_token: (see authentication)
- sig: (see authentication)
- return value:
Item types
Particles
Particles contain metadata in the top level attributes of the particle object, and content data in a data attribute. The entire particle object, when compiled with a template, produces a rendition of the content.
Auto-generated
Particles have two custom auto-generated attributes:
- tokens: (string) Facilitates the keyword-based searching of particles; composed by taking the first 50 words found in the particle that is shorter than or equal to 20 characters long (these are determined by two variables, maxWords and longestWord, in /lib/tokenize.js)
- uris: (object) A list of URI's. There will always be a data attribute, pointing to the particle's data representation. For each template that matches the particle type and version, there will also be url for the rendition, as the context attribute of the particle. If, however, the template has a subcontext attribute, the URI of that template's rendition will be populated as the subcontext-context attributes of the uris object. (this logic is contained in the addUris function in /lib/itemStore.js)
Custom
Particles have, by convention, the following custom attributes:
- type: (string) This, together with version, is used to match particles to templates
- version: (string) This, together with type, is used to match particles to templates
- static: (boolean) Static particles are not embedded in iframes. This breaks the reusability and portability function of particles, and should be removed in future. The only static particle type is pull-quote
- section: (string) Not used anymore, and should always be empty
- title: (string) The particle title, representing the particle in listings
- notes: (string) Notes to editors, for instance when a particle has an embargo date, before which it should not be used
Querying particles
- endpoint: GET /particles
- query variables:
all of the item store query variables, plus:
- query: freeform text search that matches words in the tokens attribute. This is an intersection (AND) query, in other words, all words in the query has to be found in the tokens attribute for a particle to be returned.
- return value: List of Particles, each with the following structure:
EXAMPLES UNIQUE TO PARTICLES:
http://particle-api-staging.eip.telegraph.co.uk/particles?query=government+elections (returns all particles that, somewhere in data, contains "government elections")
http://particle-api-staging.eip.telegraph.co.uk/particles?type=puff (returns all particles where attribute "type" is "puff")
http://particle-api-staging.eip.telegraph.co.uk/particles?type=puff&version=0.0.1 (returns all particles where attribute "type" is "puff" and "version" is "0.0.1")
Querying a single particle
- endpoint: GET /particles/:particleId
- return value: A single Particles, with the following structure:
NOTE:
The data attribute is only exposed on the single particle endpoint
Creating particles
- endpoint: POST /particles
- query variables: as for item store
- body: Serialised JSON:
NOTE:
author, last_modified_date, publication_date, tokens, id and uris are automatically created
- return value: as for item store
Updating particles
- endpoint: PUT /particles/:particleId
- query variables: as for item store
- body: Serialised JSON:
- return value: as for item store
Deleting particles
- endpoint: DELETE /particles/:particleId
- query variables: as for item store
- return value: as for item store
Templates
- Templates in this context refers to handlebars templates, which are used to compile each particle out to a number of output formats (also called contexts). html is a common context. cms is another, which defines a particles edit or "create" interface in the Particle CMS. When more than one template of the same context is needed (for instance, when an alternative html view is needed), the subcontext attribute is used to distinguish between them.
Custom
Templates have, by convention, the following custom attributes:
- type: (string) This, together with version, is used to match particles to templates
- version: (string) This, together with type, is used to match particles to templates
- context: (string) Determines the output context of the template (for instance html)
- subcontext: (string) Optional. Provides the ability to render more than one rendition for the same context (for instance, different html renditions)
- template: (string) The actual template.
Querying templates
- endpoint: GET /templates
- query variables: as for item store
- return value: List of Templates, each with the following structure:
Querying a single template
- endpoint: GET /templates/:templateId
- return value: A single Template, with the following structure:
Creating templates
- endpoint: POST /templates
- query variables: as for item store
- body: Serialised JSON:
NOTE:
author, last_modified_date, publication_date, tokens, id and uris are automatically created
- return value: as for item store
Updating templates
- endpoint: PUT /templates/:templateId
- query variables: as for item store
- body: Serialised JSON:
- return value: as for item store
Deleting templates
- endpoint: DELETE /templates/:templateId
- query variables: as for item store
- return value: as for item store
Articles
Articles are semi-hydrated placeholders for full articles published in AEM (the main TMG CMS), stored for the purpose of associating particles with articles they are embedded in.
Custom
Articles have the following custom attributes, determined by their representation in the main content API:
- author: (array) The default author attribute created by itemStore is overwritten by the author attribute from the content API.
- datePublished: (string) The time the article was published in the main CMS (as opposed to publication_date, which is the time when the article was ingested into the particle API. The latter should always be slightly larger (later))
- dateModified: (string) The time the article was last modified in the main CMS (as opposed to last_modified_date, which is the time when the article was last modified into the particle API. The latter should always be slightly larger (later))
- active: (boolean) This should always be true, as non-active articles don't get ingested
- status: (string) pub|draft - should always be pub, as draft articles are not ingested
- headline: (string) The article headline
- image: (array) An array of images in the article, with a useful number of renditions for each image
- channel: (string) The channel the article was published in
- keywords: (array) An array of keywords (tags) the article is tagged with
- description: (string) Always empty
- tmgId: (string) The article id in AEM
- url: (string) The article's public URL
- video: (array) An array of videos in the article
Querying articles
- endpoint: GET /articles
- query variables: as for item store
- return value: List of Articles, each with the following structure:
}
Querying a single article
- endpoint: GET /articles/:articleId
- return value: A single Article, with the exact same structure as in Querying articles above.
Creating articles
- endpoint: POST /articles
- query variables: as for item store
- body: Serialised JSON, with the exact same structure as in Querying articles above.
NOTE:
last_modified_date, publication_date, and id are automatically created
- return value: as for item store
Updating articles
- endpoint: PUT /articles/:articleId
- query variables: as for item store
- body: Serialised JSON, with the exact same structure as in Querying articles above.
- return value: as for item store
Deleting articles
- endpoint: DELETE /articles/:articleId
- query variables: as for item store
- return value: as for item store
Utilities
Handlebars compilation
Compile arbitrary data
- endpoint: POST /templates/compile/:templateId
- body: Serialised JSON - data to compile into the template
- return value: String - the result of compiling the data supplied in the POST body with the template identified by templateId
Compile particle content by context
- **endpoint: GET /particles/compile/:particleId/:context
- return value: String - the result of compiling the particle with the first template that matches the type and version of the particle identified by particleId, and that also matches context.
Compile particle content by context and subcontext
- endpoint: GET /particles/compile/:particleId/:context/:subcontext
- return value: String - the result of compiling the particle with the first template that matches the type and version of the particle identified by particleId, and that also matches context and subcontext.
Animated gif frame extraction
Single frame
- endpoint: POST /process-image/extractFrame/:frameNumber/:outputType
- body: Base64 encoded animated GIF image
- return value: Base64 encoded frame from GIF image (as image type outputType)
All frames
- endpoint: POST /process-image/extractFrame/:outputType
- body: Base64 encoded animated GIF image
- return value: Array of Base64 encoded frames from GIF image (as image type outputType)
Base64-encode image
- endpoint: GET /process-image/returnBase64/:imageUrl
- return value: Base64 encoded image (the image at the url-encoded URL imageUrl)
Finance
Get latest stock price and charts
- endpoint: GET /finance/:type/:code
- path parameters:
- type: One of ukIndex, commodity, or forex
- code: The ticker symbol
- return value:
http://particle-api.eip.telegraph.co.uk/finance/ukIndex/BARC
http://particle-api.eip.telegraph.co.uk/finance/forex/GBPUSD
http://particle-api.eip.telegraph.co.uk/finance/commodity/US@@DPL.1
Snapshot latest stock price and charts
- endpoint: POST /finance/:type/:code
- path parameters:
- type: One of ukIndex, commodity, or forex
- code: The ticker symbol
- return value:
http://particle-api.eip.telegraph.co.uk/finance/ukIndex/BARC
Legacy finance proxy API TO BE DEPRACATED
- endpoint: POST /finance/:code
- path parameters:
- code: The ticker symbol
- return value:
http://particle-api.eip.telegraph.co.uk/finance/BARC