Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add flair indexing #546

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions docs/dapp/emerald/index-emerald-onchain-data.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
Index Emerald onchain data
Real-time and historical custom data indexing for any evm chain.

[Flair](https://flair.dev) offers reusable **indexing primitives** (such as fault-tolerant RPC ingestors, custom processors, re-org aware database integrations) to make it easy to receive, transform, store and access your on-chain data.

[![flair architecture](https://imgur.com/0q5bHZK.png)](https://docs.flair.dev/)


## Why Flair?

Compared to other alternatives the main reasons are:

* 🚀 Adopting **parallel and distributed processing** paradigm means high scalability and resiliency for your indexing stack. Instead of constrained sequential processing (e.g Subgraph).
* 🧩 Focused on **primitives**, which means on the left you plug-in an RPC and on the right you output the data to any destination database.
* 🚄 Native **real-time stream processing** for certain data workload (such as aggregations, rollups) for things like total volume per pool, or total portfolio per user wallet.
* ☁️ **Managed** cloud services avoid DevOps and irrelevant engineering costs for dApp developers.
* 🧑‍💻 Avoid decentralization **overhead** (consensus, network hops, etc) since we believe to enable best UX for dApps reading data must be as close to the developers as possible.
### Features

* ✅ Listen to **any EVM chain** with just an RPC URL.
* Free managed RPC URLs for +8 popular chains already included.
* Works with both websocket and https-only RPCs.
* ✅ Track and ingest **any contract** for **any event topic.**
* Auto-track new contracts deployed from factory contracts.
* ✅ **Custom processor scripts** with Javascript runtime (with **Typescript** support)
* Make external API or Webhook calls to third-party or your backend.
* Get current or historical USD value of any ERC20 token amount of any contract address on any chain.
* Use any external NPM library.
* ✅ **Stream** any stored data to your destination database (Postgres, MongoDB, MySQL, Kafka, Elasticsearch, Timescale, etc).

## Getting Started

1️⃣ Clone the [starter boilerplate](https://github.com/flair-sdk/starter-boilerplate) template and follow the instructions

```bash
git clone https://github.com/flair-sdk/starter-boilerplate.git
# ... follow instructions in README.md
```
:::info

Boilerplate instructions will create a **new cluster**, generate **an API Key**, and set up a manifest.yml to index your **first contract** with **sample custom processor** scripts.

Learn more about the [structure of manifest.yml](https://docs.flair.dev/reference/manifest.yml).

:::

2️⃣ Configure Emerald RPC nodes

Set a unique namespace, Emerald chainId and RPC endpoint in your config. Remember that you can add up to 10 RPC endpoints for resiliency.

```yaml
{
"cluster": "dev",
"namespace": "my-awesome-emerald-indexing-dev",
"indexers": [
{
"chainId": 42262,
"enabled": true,
"ingestionFilterGroup": "default",
"processingFilterGroup": "default",
"sources": [
# Highly-recommended to have at least 1 websocket endpoint
"wss://emerald.oasis.dev/ws",
# You can put multiple endpoints for failover
"https://emerald.oasis.dev"
]
}
]
}
```

3️⃣ Sync some historical data using [backfill command](https://docs.flair.dev/reference/backfilling). Remember that `enabled: true` flag in your `config` enabled your indexer to capture data in real-time already.

```bash
# backfill certain contracts or block ranges
pnpm flair backfill --chain 42262 --address 0xd8a2ae03f6edd58999a0f1005db7a6532f2aa79e -d backward --max-blocks 10000

# backfill for a specific block number, if you have certain events you wanna test with
pnpm flair backfill --chain 42262 -b 7260447

# backfill for the recent data in the last X minute
pnpm flair backfill --chain 42262 --min-timestamp="30 mins ago" -d backward
```


4️⃣ [Query](https://docs.flair.dev/#getting-started) your custom indexed data.

5️⃣ Stream the data to your [own database](https://docs.flair.dev/reference/database#your-own-database).

## Examples

Explore real-world usage of Flair indexing primitives for various use-cases.

### DeFi

* [Aggregate protocol fees in USD across multiple chains](https://github.com/flair-sdk/examples/tree/main/aggregate-protocol-fees-in-usd)
* [Calculate "Health Factor" of positions with contract factory tracking](https://github.com/flair-sdk/examples/tree/main/health-factor-with-factory-tracking)
* [Index Uniswap v2 swaps with USD price for all addresses](https://github.com/flair-sdk/examples/tree/main/uniswap-v2-events-from-all-contracts-with-usd-price)

### NFT

* [Index ERC721 and ERC1155 NFTs on any EVM chain with an RPC URL](https://github.com/flair-sdk/examples/tree/main/erc721-and-erc1155-nft-indexing)

## Need help?

[Our engineers](https://docs.flair.dev/talk-to-an-engineer) are available to help you at any stage.
1 change: 1 addition & 0 deletions sidebarDapp.js
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ const sidebars = {
items: [
'dapp/emerald/writing-dapps-on-emerald',
'dapp/emerald/integrating-band-oracle-smart-contract',
'dapp/emerald/index-emerald-onchain-data',
],
},
{
Expand Down