Understanding Bacalhau 1.0: Unlocking the Potential of Private Data
Source: Filecoin Network
This article is based on a speech by Simon Worthington at the Boston Summit in May 2023. Watch the video (in Chinese).
Bacalhau is revolutionizing data processing by supporting on-site computation. Instead of moving data to the code, it sends the code to where the data is located for analysis. By preserving data and allowing authorized access, auditing, and controlled computation, more data can be used while reducing the risk of misuse. This is the answer to data governance problems. The growth rate of data surpasses network bandwidth by 45%, and 57% of data is stored outside of the cloud or traditional data centers. For any large-scale organization, moving data is slow and costly.
There is another good reason to keep data local: control. Nearly 100% of data is subject to some form of governance, whether it's through mandatory regulations like the Health Insurance Portability and Accountability Act (HIPAA) or the General Data Protection Regulation (GDPR), or for protecting sensitive financial or company confidential information. Transferring data to computation takes it out of its usual secure zone and increases the risk of abuse.
Most data is not strictly open or closed but exists within a certain scope. Within this scope, specific access permissions can be granted for specific purposes.
Source: The ODI
Since 2008, the total amount of fines for data governance worldwide has reached nearly $250 billion. Therefore, it is not surprising that most companies are afraid of sharing data, which also leads to 68% of enterprise data being undeveloped and unused. In fact, most controlled data can theoretically be shared and used for more effective decision-making - but it requires the right people and the right purposes.
Data sharing requires technical enforcement
Most organizations attempt to meet this requirement through strict data sharing agreements or contracts. These agreements are expensive and time-consuming - for enterprises such as government or financial institutions, it can take months to achieve data sharing among internal teams through data governance.
What's worse is that these agreements don't work at all - most data sharing agreements are completely unenforceable and only provide a false sense of security. Once the data crosses the trust boundary, only soft mechanisms (such as trust that everyone will comply with the agreement) can prevent misuse. The actual operation of sharing data is invisible and difficult to regulate.
"It has been shown that contracts or agreements between data providers and data users often do not work.
In the Cambridge Analytica scandal, contractual terms were completely ignored and personal data was abused.
Lack of any compelling technical evidence can make it difficult for courts to obtain valid information and make it difficult for regulatory agencies, politicians, journalists, and the public to understand what has happened."
- "Putting the trust in data trusts," Register Dynamics, 2019
Clearly, what we need is a new method of reusing data across trust boundaries: through this method, analysts can access data simply and controlled, while data owners will not face the risk of regulatory fines and making headlines.
Bacalhau makes data sharing transparent and auditable
At Bacalhau, we believe that localized data computation is the answer to solving the challenges of data governance. By preserving data and allowing authorization, auditing, and controlled computation, more data can be used while reducing the risk of misuse.
More importantly, because Bacalhau is a distributed computing platform, there is no need to transfer data to a central storage. Data can be stored anywhere it should exist within the organization, avoiding difficult organizational changes and not depriving data owners of any control.
With great pride, we announce that as part of Bacalhau 1.0, we have added job and data governance capabilities. With Bacalhau, data owners can control who, what, where, why, and how their private data is computed.
Bacalhau Governance Code and Output
Bacalhau uses a two-step process for job governance. Firstly, data owners have the opportunity to review whether a job complies with their policies. This pre-governance stage occurs before the job starts running and allows governance personnel to approve or deny computation based on the data to be used, the requestor of the job, and the code to be executed for that job.
While humans always remain in control, not every decision needs to be made by humans. The pre-governance process is highly flexible and can be automated as desired. Data owners can set policies, deep-check computations about to run, set different policies for different individuals, and invoke sophisticated algorithms for security and risk analysis. When a task is not suitable for automated governance, a final decision can be made manually.
Bacalhau provides two gateways for computation, one before computation and one after computation.
Once approved, Bacalhau will send the job to the appropriate executor, which can only access the requested data and is isolated from the host system's security. Bacalhau imposes resource constraints on jobs to control processing power and memory usage.
Although job moderation provides a reasonable first line of trust defense, it is generally challenging to determine what it will do without running the computer program, which requires technical skills. The UK Office for National Statistics and other controlled research environments have been safely allowing controlled access to data for decades, and we have drawn on their experience and practices in the digital field. Therefore, in addition to pre-controls, Bacalhau also allows modifications to the results after execution, before releasing them to the job submitter.
When Bacalhau completes the computation, it saves the results to a private pre-release area. The administrator then checks the results based on the context of the job to determine if they are the expected results. If the administrator determines that the content is suitable for sharing, they can download the results. More importantly, access to the private storage area is strictly locked, and users can only stream the results to their own job using Bacalhau's download function.
Like job moderation, the results can undergo a full set of complex analysis. With the Amplify technology, data owners can automatically detect Personally Identifiable Information (PII), summarize tabular data in CSV format, and analyze content in images and video clips. The generated metadata can be used for automatic result release and provide valuable information for human decision-making.
Moderation Opens up a New Era of Federated Learning
Computing on data separated by trusted boundaries allows for extensive data sharing, but there is currently no secure technical solution. If an organization's data can generate common value through wider sharing, these organizations can now apply Bacalhau job moderation and open up data access without complex data governance.
For example, a university can provide more data to citizen scientists or external researchers, a government department can allow another department to analyze its data, or a highly regulated financial institution team can allow another team to conduct in-depth analysis of its data. In conclusion, it is important not to publish raw data to users with lower trust levels. Bacalhau ensures that users get their analysis results and nothing more.
The same distributed controlled computing model can also achieve federated learning among participants from different organizations. With Bacalhau, independent organizations can perform in-depth analysis on aggregated data without the need to share the data. With federated learning techniques, data scientists can now train machine learning or AI models on datasets from many different independent or even competitive organizations, without these organizations losing control of their data and being able to accurately see how the data is being used.
For example, central government agencies responsible for formulating macro policies can leverage the data held by local organizations. Similarly, industry organizations such as insurance regulatory agencies can train models by submitting federated learning Bacalhau tasks to all their member insurance companies.
If the dataset is centralized, there is a high risk of selling or abusing the valuable aggregated data; but if the data is kept locally, each insurance company can ensure that their data is only used for mutually agreed upon beneficial purposes.
Computing Islands for Topic-specific Analysis
Finally, the fine-grained control over job execution provided by Bacalhau now allows administrators to become gateways to access computing islands. In this structure, independent computing providers and data owners interested in offering resources for specific purposes can delegate job authorization to trusted controllers.
For example, scientists who have collaborated to collect medical data for the treatment of cancer can provide the data and compute through external controllers they trust. The controllers only accept jobs that comply with agreed-upon policies - in this case, only jobs that contribute to new cancer treatments are allowed.
By delegating external access requests to controllers, scientists can focus on larger public interest goals. With Bacalhau's powerful audit logs, scientists can verify in the future whether controllers have acted according to agreed-upon policies.
Bacalhau is the Future of Data Sharing
We are excited to announce the job and data control features in Bacalhau 1.0! We believe that data computing represents a new approach to data sharing - in short, ensuring data security by not sharing the data!
Nowadays, some companies and government agencies have recognized the potential of controlling computing across trust boundaries, and we are working with them to explore it. If you want to learn more about how these features can benefit you, join the Bacalhau Slack or get in touch with us directly.
