Categorygithub.com/guer-co/hackfs-mdcfrontend
package
0.0.0-20200808230610-ee314763e6cf
Repository: https://github.com/guer-co/hackfs-mdc.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
8
Dependents
0
Files
68 SLOC

# README

P^3 PROTOCOLS

MICROSERVICES BRINGING DECENTRALIZED CONTENT TO THE WORLD

Introduction

We're building modular, independent microservice protocols allowing web developers, content producers, and consumers to mutually benefit from the advantages of emerging technologies in Web3 / Decentralized Computing. Functionally, we provide:

Payments & Paywall:

Frictionless, integrated payments for consumers using cryptocurrencies and smart contracts with innovative revenue modeling

Advanced Distribution and Licensing:

Providing content producers more secure methods to legitatemely distribute their work using encryption and emerging technologies

Content Authentication:

Cryptographically authenticated copies help ensure consumers receive untampered content, and producers can verify if a user should have access to that copy

Take me right to the Development Area!

Context

Cryptocurrencies as payment for subscriptions is not enough to fundamentally improve relationships between digital platforms, content producers, and content consumers. While theoretically "programmable money" has obvious advantages over credit cards, in practice, it does not provide sufficient utility over existing systems to warrant overcoming unfamiliarity. We believe a systematic approach is required for adoption, where several microservices can collectively provide greater utility over existing systems, while remaining modular enough to provide developers and communities the flexibility they desire to address their needs. In this section, we want to highlight advantages and motivations of adopting our Web3 protocols.

Misaligned Incentives in Revenue for Content

Marc Andreesen has been quoted lamenting the inability to build payments natively into the Web, as early as Netscape and other early browsers. As a result, we have become dependent on a limited range of successful revenue models for digital businesses. Among these:

Ad-based Revenue:

Content published on the web relying on advertisements to support the production of content. This has led to everyone from legitimate news publications to aspiring micropublishers like bloggers optimizing content and design to encourage Click-Through Rates rather than truthfulness and great user experience.

End-user as a Product:

As an evolution of ad-based revenue, platforms have developed to monetize user behavior, selling end-user activity to advertisers as big data, often at the expense of privacy. Sophisticated techniques, often utilizing machine learning, provide "tailored experiences" in an attempt to increase value to advertisers with little regard to their user base. This is now commonplace in social networks, and has contributed even to political manipulation on a global scale.

Subscriptions:

Paying for bundled content for limited periods of time inappropriately incentivizes producers to decide between quality and quantity of content. Often, "publications" sacrifice quality to ensure a desirable quantity or frequency. End-users have no guarantee to continued access for content they have purchased.

These behaviors intend to cover costs, such as the salaries of producers, developing and managing platforms, and storing and distributing that content. By embracing existing Web2.0 revenue models, and enabling Web3.0 models, we expect incentives to better align with organizational goals and better end-user experience.

Challenges in Distribution and Digital Rights Management

Digital content has always faced issues with enforcing licensing in distribution, only exacerbated by the next-to-nothing cost of duplication. With Web2.0, we identify the following challenges in distribution:

Going Viral:

Having content "go viral", or reach a wide audience, is extremely beneficial in models that prioritize volume. Often times this requires a complete forfeiture of control and custody of content, in which case producers do not capture value or has visibility to true reach.

Paywall-only:

Alternatively, by offering all content behind a paywall, producers dramatically limit their reach and capacity for organic reach, but capture all value. This often discourages new consumers, and incentivizes non-paywall producers to offer substitutions.

Static Permissions:

Certain content value capture is limited by the inability to enforce different permissions policies at different points in its lifecycle. At initial release, a producer may want the widest possible audience, but then be able to monetize after a period of time, e.g. Pre-web introductions of pop songs via radio, with at-home sales after building demand.

Authenticating Content

Manipulation of content:

Content that is easily accessed can be manipulated and redistributed as original with no evidence that it has not been tampered with.

Biased Referencing:

Content behind paywalls is often referenced by ad-supported publications which cannot be verified by consumers without purchasing original content. By enforcing controls over content, content producers can better enforce undesired referencing.

Secondary Distribution:

Similar to biased referencing, content can be re-published by third parties and cited, capturing value from the original content producer.

Other Considerations native to Web3

  • Censorship-resistant & Immutable
  • Immutable
  • Lower storage costs with comparable redundancy and availability
  • Trustless name resolution and hosting

Solutions

Emerging Technologies

In order to solve for these existing Web2.0 challenges, we are employing emerging Web3.0 technologies, including:

  • Ethereum, including ERC20 tokens for payment, smart contracts for access management, and EC-keypair as the basis of digital identity and encryption

  • Filecoin, in conjunction with IPFS for decentralized file storage

  • Textile, to facilitate file storage, store metadata, and serve as a Filecoin Gateway

  • LibP2P, to facilitate P2P communications

  • Networked Trusted Execution Environments (TEE's) for baremetal security and trustless virtualization

  • Unstoppable Domains for decentralized hosting for the front-end

  • Fleek, as a front-end CMS

  • ENS as an Ethereum-native Name Service

New Revenue Models

We're introducing new revenue models with the intention to disincentivize click-bait, ad-driven design, sensationalist content, content reappropriation, and allow non-content producing third parties direct relationships with consumers in place of content producers. For users, the result should be content-first, ad-free design, with several options to frictionlessly support and share content.

Using these technologies, our protocols enable the following novel revenue models for content stored on Web3:

Frictionless Micropayments

Consumers can support content producers directly, either within the browser or a native content app, by purchasing content as-you-go. Using meta-transactions, consumers can set a budget for a set period of time, and frictionlessly navigate through content discovery to payment in a familiar environment. Content can be rented for one-time viewing, or purchased for perpetual access.

Sponsored Consumption

Building on the concept of frictionless microtransactions, a business or organization can sponsor or subsidize a consumer's content browsing, providing either a stipend or just-in-time payments in exchange for access to their browsing history. This allows third-parties to establish relationships directly with willing consumers, creating more relevant data upon which to base targeted, programmatic advertising. Content Producers, free of the constraints of traffic for advertisements, can maximize the quality of their content.

Subscriptions

Content consumers who regularly enjoy content created by either individual producers, or collective producers in the form of a publication, and purchase time-based access in the form of a subscription. This can be customized by the producer or publication to be either time-based access, where at the conclusion of an active subscription, a consumer loses all access, or purchased access, where a consumer retains access for all content produced within a time-frame with active subscription. 
Using meta-transactions, a consumer can easily and frictionlessly subscribe and renew to content without ever reaching for a credit card, ensuring an improved but familiar experience. Ongoing subscriptions can also be subsidized by third parties.

Pre-paid Content Sharing

Consumers or organizations wishing to share premium content with others can elect to "gift" content or subscriptions to other known addresses. In doing so, other consumers can be added to content for either one-time or perpetual viewing.

Conditional Free

In order for producers to reach the widest possible audience, they may elect to release content as "Free". Unlike traditional free media or content, when paired with P^3's distribution protocol, free content can be easily made conditional. For example, content can be free for first view, or for a limited amount of time, with any access beyond the initial conditions requiring some form of payment. In this sense, unconstrained content still has an opportunity to "go viral" without compromising the value or future compensation of that content. Important, public-good breaking news can be widely distributed for free, but still be monetized when requested for archival purposes.

Innovation in Distribution

Controlled distribution of content, complete with revocable/irrevocable access, time-based access, permanent ownership of digitally unique content, and transferable access, is critical to enabling next-generation DRM and monetization. 
To that end, we have devised a systemic distribution overview with both content producers and consumers in mind, providing producers full control over the distribution of their content while equally guaranteeing consumers perpetual license to purchased content. 
This systematic "goldilocks" condition is achieved through the introduction and integration of several independent, decentralized, and immutable systems which require little-to-no participation from either consumers or producers, beyond 2-3 clicks. 
Broadly speaking, user and access management is handled independently using smart contracts on EVM to authenticate users, content, and payment. Transport/Session layer security is conducted in a completely decentralized, trustless manner. Prior to decentralized storage, a decentralized security service leveraging Trusted Execution Environments (TEE's) manages encryption/decryption for storage, and temporary re-encryption for sessionable distribution. P2P transportation protocols ensure secured delivery across networks, while reducing dependencies on third parties such as CA's and DNS Resolvers.

User & Access Management

Smart-contract based user management ensures immutability and correctness for both content producer and consumer. By using an instance-based access management system, we provide developers the granularity to build custom management systems for their platform with one microservice.

Session/Transport Layer Security Session-layer security built natively for Web3. Unlike TLS or other transport layer security schemes, dSLS leverages EC keys and networked SGX Trusted Execution Environments to generate and distribute session symmetric AES128 keys for secure P2P transmission of content.

Storage-side Encryption

Serverless, decentralized trusted execution environment facilitates uploads and requests for content between decentralized storage networks and content-consumers. This service receives encrypted content from decentralized storage networks, deterministically reconstitutes the the encryption key using MPC, returns content to plaintext, and re-encrypts it with a session key for distribution. This service allows content producers to better control distribution to consumers by limiting their access to in-browser or in-app sessions. At the termination of the connection, a new key would be required to access content.

Decentralized Storage

We leverage Textile buckets and threadsDB to manage content stored on Filecoin/IPFS.

P2P Transportation (Including initial TLS connection to Web3)

End-to-End Authentication Integrity

For both content producers and content consumers, verifying the authenticity of distributed copies is critical to the success of monetizing digital content. Cryptographically authenticating copies through several layers of encryption and transport channels becomes essential for both parties. We intend to achieve this by creating a Hash-based Message Authentication Code at upload, which serves as the root throughout the content's lifecycle. With every encryption/decryption, a new HMAC is created, referencing the original, and can be cryptographically signed to origin. This provenance could likely be replaced by a succinct zero-knowledge proof. By associating HMACs with content and encryption, we could also better empower referencing in content, by verifying authenticity of referenced content. We acknowledge this does not solve "the last mile" problem of content distribution; users can still take screenshots, record their desktop, pdf articles, etc. We believe that this is a substantial step in the right direction, however. Root HMAC Distribution HMAC Reference HMAC ZK-Proofs P^3: Protocol Technical Specification

We recognize that Web3.0 cannot be "one-size-fits-all", and encourage dApp developers to integrate these microservices to best suit their needs. Their current functionality is intended as infrastructure primitives, which can be specialized to suit the needs of the application.

System Context

Payments

Technical Integration INSERT DIAGRAM OF PAYMENTS SMART CONTRACT INTEGRATION WITH ACCESS MANAGEMENT, dSLS, DISTRIBUTION, STORAGE, AND FRONT END* Distribution Authentication

User Experiences

frontPage - Journalism FrontPage is a content discovery, publication and monetization platform built for Web3. FrontPage seamlessly connects users to content on the decentralized web with a familiar user experience. With integrated payments, the ability to digitally own content purchased, and to verify content's authenticity, users never need to see an advertisement again.

Features

Immutability Digitally Unique Ownership Complete control over distribution Real-world Payment systems Content Authentication Seamless user experience censorship resistance monetizable control over distribution

Built with

Development Area

Blockchain

  • install Truffle: npm install -g truffle
  • install Ganache (local ethereum blockchain): https://www.trufflesuite.com/ganache and run it
  • BUILD: cd /hackfs-mdc truffle build --network development --reset

Frontend

  • cd hackfs-mdc/frontend
  • npm i
  • npm run dev

Backend

A libp2p based micro-service node, which provide service for storing, encrypting and decrypting contents.

Install protoc

https://github.com/protocolbuffers/protobuf/releases/tag/v3.12.3

Install ffmpeg

https://ffmpeg.org/download.html

ENV

export TEXTILE_HUB_USER_KEY=XXX
export TEXTILE_HUB_USER_SECRET=XXX
export TEXTILE_DB_THREAD_ID=XXX
export TEXTILE_TEST_DB_THREAD_ID=XXX

Start the server node

cd cmd/server
go generate
go build
./server
2020-07-20T08:12:15.268-0700	INFO	server	server/server.go:37	host.ID: QmeJ4nc8NZSiJhAM4gPA6chzdhjMFzhC51j5WLz5XA5pJn
...
2020-07-20T07:14:25.268-0700	INFO	common	server/server.go:80	Announcing this server node with rendezvous string: pay3-rendezvous-01EDC2XSADF7CRB5AJ6SCNWHCG

Start the proxy client

  • On another terminal, start the proxy client with the "rend" value shown above i.e. pay3-rendezvous-01EDC2XSADF7CRB5AJ6SCNWHCG
  • It would take a few sec to discovery the server node, then starting the httpproxy.
cd ../proxyclient
go build
./proxyclient -rend [rend output from server]
...
2020-07-22T08:13:39.977-0700	INFO	common	proxyclient/proxyclient.go:78	Searching for server node with rend: pay3-rendezvous-01EDVHC0XY5G0JKEVM8C79N95Q
2020-07-22T08:13:46.893-0700	INFO	common	proxyclient/proxyclient.go:90	serverAddrInfo: {QmcnVnqUQpbjWGbnm2LUEeFFQLiCeywysWm3rLMa2k4DNZ: [/ip4/127.0.0.1/tcp/60327 /ip4/192.168.86.193/tcp/60327 /ip6/::1/tcp/60328 /ip6/fe80::4088:f00d:bb5:5545/tcp/60328 /ip4/99.102.91.69/tcp/60327]}, p: {QmcnVnqUQpbjWGbnm2LUEeFFQLiCeywysWm3rLMa2k4DNZ: [/ip4/127.0.0.1/tcp/60327 /ip4/192.168.86.193/tcp/60327 /ip6/::1/tcp/60328 /ip6/fe80::4088:f00d:bb5:5545/tcp/60328 /ip4/99.102.91.69/tcp/60327]}
start go server
2020-07-22T08:13:46.893-0700	INFO	common	httpproxy/httpproxy.go:115	httpproxy running at :8888

Upload test

curl -X POST http://localhost:8888/api/ipfs \
    -F "file=@../../pkg/textilehelper/test01.png" \
    -H "Content-Type: multipart/form-data"

curl -X POST http://localhost:8888/api/ipfs \
    -F "file=@../../pkg/tools/samplemedia/testVideo01.mov" \
    -H "Content-Type: multipart/form-data"

curl -X POST http://localhost:8888/api/upload \
    -F "file=@/Users/sing.yiu/Playground/hackfs/hackfs-mdc/backend/pkg/textilehelper/test01.png" \
    -F "ownerId=testowner02" \
    -F "description=testdescription02" \
    -H "Content-Type: multipart/form-data"

return
{"ownerId":"testowner02","fileName":"test01.png","fileType":"image","fileSize":500269,"description":"testdescription02","threadKey":"bafkqyvemctry6u5zbmzdmws5ubgwnnmitniribu7xe2qudfvhaqffka","bucketKey":"bafzbeihebjpoa6cj7j4nuqidgjxdatqep5f54nfta3oq543ij26y7i453u","encryptedUrl":"https://hub.textile.io/ipns/bafzbeihebjpoa6cj7j4nuqidgjxdatqep5f54nfta3oq543ij26y7i453u","previewUrl":"https://hub.textile.io/ipns/bafzbeics7vq3bg4tufogmczwtzsdb5cfvbxc3lz7ismpom3bqpagnslpry/thumbnail.jpg","receivedAt":1595814840773,"updatedAt":1595814843631}

Download test

http://localhost:8888/api/download/testrequester01/bafzbeibbpbqs6oizlyg7dh7tkjxmlldp3l2xdg5yghbtw4ehxl2xiwkyba

Database API

  • JSON query: i.e. query all ContentData records with field "ownerId" equals "testowner02"
  • N.B. the field "jsonInput" where the query parameters go into, is an escaped Json String !
curl -i -k -H "Content-Type: application/json" -d '{"requesterId":"testRequester01", "type":"query", "jsonInput":"{\"collection\":\"ContentData\", \"fieldPath\":\"ownerId\", \"operation\":\"Eq\", \"value\":\"testowner01\"}"}' http://localhost:8888/api/json

return
{"messageData":{"clientVersion":"pay3-server-node/0.0.1","timestamp":1595816336,"id":"01EE711Q5KJGJ2BB58SER22WJ0","nodeId":"QmR4adopEs97RKS737SCaT6jfCaPExbAN2ohE7nzhyLyLu","nodePubKey":"CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC51cbCBoEhvZt61FmlGgWcbS7A3jPJuWRY0RZN8UKDkn79vkSubywbZQ4QUh9ivsKiIV82UE5p8qsJ7dtEFN1Jde+u3PtEeLhwLb5iwqumUO80Kjg6bkWsGG2v1vyAGu3VqqVNTmQ/JQ7xQ6yhcIXleFM61+wXHJMa9p5EA0dh9SAjILBHnlZIQqzsRDjLqWdaDwYiP96DltiGNtj3RhxTuHgUCHSJF5FI4x9dQMpAewqsdwhZC461g8RGMqUpopLDhflyZWLfzmLQ5XP6Xcl9EiwsB2kG8Dn/XBLeq1qDZlM6A8XVKL8MRGT8oybUOcNHkhnfkFwLExe3cC6AQeZzAgMBAAE=","sign":"M1ZvcQDIM50Iooh2ulnaeciR3Z2yQ1PnyXW4Tr2xw6wcN8RF1g0i7keRHakVm36WYnfNvm1nPDeyckYCOXRdbB3WBxQc8JUoj4I2/v01Z9BgwXrXK+Nk1E1rY7Gc/yzP3iSZPv4QGgRKAUIiBtQcp2w+lEcXUoSQCtV449swpJbo1mw1jwU5dplKtlt+eoDMmAHke4V1U4ZKrUzkCZMl0YU7BmHSlR/eC+EMPKrxdrJrZwhX6eg4FraMFLHJ03t3w5jLAv5E13PYPi+TvJ7JW+Yw7Un/zgV/kjHRlWnpAss+KM0zx9798YjXt/S8ZYWZAAPfPnsr28a+0FGqvRl3Fg=="},"responseData":{"code":200},"jsonData":{"requesterId":"testRequester01","type":"query","jsonOutput":"[{\"ownerId\":\"testowner01\",\"fileName\":\"test01.png\",\"fileType\":\"image\",\"fileSize\":500269,\"description\":\"test01\",\"threadKey\":\"bafkqyvemctry6u5zbmzdmws5ubgwnnmitniribu7xe2qudfvhaqffka\",\"bucketKey\":\"bafzbeig57kazgtszsil3vjk64ccgmtf34hqu43yqzdzzgqu45dy3edi3bu\",\"encryptedUrl\":\"https://hub.textile.io/ipns/bafzbeig57kazgtszsil3vjk64ccgmtf34hqu43yqzdzzgqu45dy3edi3bu\",\"previewUrl\":\"https://hub.textile.io/ipns/bafzbeidfjtnkf6dp472oxixfsdxksyivrysulmlomgfyq5gqumuuvxj5ui/test01.png\",\"receivedAt\":1595254379594,\"updatedAt\":1595254383095},...]","receivedAt":1595816336565,"updatedAt":1595816336728}}