No Cost Data Scraping With GitHub Actions And Neo4j Aura

1

Lobsters

Name: Lobsters Data Import On: push: paths: - .github/workflows/lobsters.yml workflow_dispatch: schedule: - cron: '*/60 * * * *' Jobs: scheduled: runs-on: ubuntu-latest steps: - name: Check out repo uses: actions/checkout@v2 - name: Fetch newest uses: githubocto/flat@v2 with: http_url: https://lobste.rs/newest.json downloaded_filename: newest.json - name: Neo4j import uses: johnymontana/flat-graph@v1.2 with: neo4j-user: ${{secrets.NEO4J_USER}} neo4j-password: ${{secrets.NEO4J_PASSWORD}} neo4j-uri: ${{secrets.NEO4J_URI}} filename: newest.json cypher-query: > UNWIND $value AS article MERGE (s:User {username: article.submitter_user.username}) ON CREATE SET s.about = article.submitter_user.about, s.created = DateTime(article.submitter_user.created_at), s.karma = article.submitter_user.karma, s.avatar_url = "https://lobsete.rs" + article.submitter_user.avatar_url MERGE (i:User {username: article.submitter_user.invited_by_user}) MERGE (i)<-[:INVITED_BY]-(s) MERGE (a:Article {short_id: article.short_id}) SET a.url = article.url, a.score = article.score, a.created = DateTime(article.created_at), a.title = article.title, a.comments = article.comments_url MERGE (s)-[:SUBMITTED]->(a) WITH article, a UNWIND article.tags AS tag MERGE (t:Tag {name: tag}) MERGE (a)-[:HAS_TAG]->(t).

#Social Networks #Social News #Knowledge Sharing 120 social mentions

2

GitHub Actions

When working with data a common task is fetching data from some external source on a recurring basis and importing into a database for further analysis or as part of our application. Setting up servers to handle this can be time consuming and error prone. I recently came across a workflow using GitHub Actions and Neo4j Aura that makes this a breeze and with the free tiers of both GitHub Actions and Neo4j Aura is free to set up and run forever - great for side projects!

#DevOps Tools #Continuous Integration #Continuous Deployment 272 social mentions