Proposal: Enhanced User Logging System on Akash Console

[HoomanDgtl]

Proposal: Log Forwarding Integration for the Akash Network: Empowering User-Controlled Logging Solutions

Introduction

This proposal represents a strategic pivot in our approach to implementing logging solutions for the Akash Network. Following in-depth discussions with the Akash core team, we have refined our strategy to better align with the network's principles of decentralization and user needs while maintaining simplicity and effectiveness.

Initially, we proposed developing a self-hosted logging infrastructure using Grafana, Prometheus, and Loki. However, discussions with the core team highlighted the following key insights:

• Many users already have established relationships with various logging providers.
• Concerns were raised about overloading provider services for streaming logs.
• Leveraging existing battle-tested solutions provides immediate value.
• Multiple Akash customers prefer SaaS logging solutions like Datadog and Grafana Cloud.

Based on these insights, we revised our approach to focus on integrating robust log forwarding capabilities, allowing users to utilize their preferred logging solutions while maintaining full control over their data and costs.

---

Problem Statement

The Akash Network console currently lacks a persistent logging system, leading to significant challenges:

• Limited Log Availability: Only live logs are accessible; there is no historical retention, making debugging difficult after a system crash or unexpected behavior.
• Ineffective Debugging: Without log history, identifying the root cause of issues is challenging, especially for complex deployments.
• Community Support Barriers: New users or developers lack the ability to share logs for troubleshooting within the community, complicating the onboarding process.

---

Proposed Solution

We propose integrating comprehensive log forwarding capabilities into the Akash console to:

• Enable persistent log storage through users' preferred logging solutions.
• Provide a simple, configuration-based setup process.
• Allow users to maintain control over their logging costs and retention policies.
• Leverage existing, battle-tested logging infrastructure.

---

Implementation Details

Console Integration

Settings Page Enhancement:

• Add a new "Logging Configuration" section to the console settings.
• Allow users to input credentials for supported logging destinations.
• Validate credentials before saving.
• Securely store logging configuration.

SDL Builder Integration:

• Add an "Enable Logging" checkbox in the SDL builder section.
• Automatically inject the logging sidecar configuration into the deployment when enabled.
• Allow users to modify logging parameters through the UI.

---

Technical Implementation

Logging Sidecar Service:

• Automatically add a logging sidecar container to deployments when logging is enabled.
• Configure the sidecar to collect logs from all containers in the deployment.
• Forward logs to the specified logging destination using stored credentials.
• Implement retry mechanisms and buffering for reliable log delivery.

Supported Logging Destinations (Initial Phase):

• Datadog
• Grafana Cloud
• Elastic Logstash
  (Expand support to additional platforms based on user feedback.)

---

Implementation Timeline (2-3 Months)

Phase 1: Frontend Development

• Design and implement console UI components.
• Add a logging configuration section in the settings page.
• Develop credential management and validation workflows.
• Create a logging toggle and basic configuration options in the SDL builder.

Phase 2: Backend Development

• Design and implement secure credential storage.
• Develop the logging sidecar container configuration.
• Create a system for injecting logging configurations into deployments.
• Implement initial logging destinations.

Phase 3: Testing and Documentation

• Conduct end-to-end testing of logging integration.
• Create user documentation and troubleshooting guides.
• Develop documentation for adding support for additional logging destinations.

---

Expected Outcomes

This revised logging integration approach will provide:

• Professional-grade logging capabilities through users' preferred solutions.
• Simplified deployment monitoring and troubleshooting.
• Cost-effective solutions leveraging existing infrastructure.
• Improved developer experience and debugging capabilities.
• Enhanced support for production workloads.

This solution aligns with Akash Network's principles of decentralization and user empowerment while providing immediate value to the community.

---

Budget Estimate

The following budget reflects the comprehensive scope of work required:

Category	Hours	Description
Research & MVP Testing	150	User needs identification and MVP conceptualization.
UI/UX Design	80	Designing user-friendly configuration flows.
Frontend Development	150	UI components and SDL builder enhancements.
Base Container Development	180	Building a robust logging sidecar container.
Data-Logging Integration	120	Integration with Datadog, Grafana Cloud, and Elastic Logstash.
Production Testing	100	Thorough testing, documentation, and refinements.

Total Hours: 780
Hourly Rate: $100
Total Budget: $78k

This budget ensures a professional-grade implementation of log forwarding integration, addressing user pain points while aligning with Akash Network's values.

Governance Timeline

• Discussion & Feedback: ongoing
• On-Chain Voting: Targeting Thursday/Friday this week

Link to POC Video: click here
Previous Prop & plan (just for record): click here

---

Thank you for considering this proposal. We welcome your feedback and community support on our updated plan. If you have any questions, please feel free to ask below.

Thank you

[anilmurty]

Hey @HoomanDgtl (Piyush and Chai),

Thanks very much for initiating this and for coming up with the POC and presenting it to the core team. To echo what you said above, not being able to capture, retain and query past logs easily is a very acute painpoint for our customers and solving it is one of 3-4 top level initiatives on the Console Roadmap (alongside CC payments, Alerting/ Notifications and better/ easier workflows for non-container native developers).

To recap our discussion (between the core team and your team):
While this solution is great - there are a couple challenges with storing logs on Akash providers:

1. Data stored on Akash providers is only persisted for the life of the lease today (because Akash is primarily a compute network/ cloud and not a storage network/ cloud). So if the lease that is used to store the logs is closed the user logs are lost which defeats the purpose of logging.
2. Multiple customers of Akash have directly indicated that they prefer using SaaS logging solutions like Datadog and Grafana Cloud. This is likely driven by the fact that many of these customers use traditional (centralized) cloud services before considering or moving to akash and they would not want to disrupt all their workflows (and monitoring solution is a key part of those workflows). So it is essential that the first tenant logging solution Akash implements support "log forwarding" to those common SaaS logging services, so as to ease the transition for customers coming to Akash from traditional clouds.
3. While you can technically store the logs AND ALSO forward them to a SaaS logging service, that would be unnecessary extra cost for the user/ customer.
4. There are concerns from the core team about overloading provider services for streaming logs. The last thing we want to do is cause provider services to have issues, because it would impact compute workloads, bid engine etc.

With those things in mind, the recommendation from the core team is to:

1. Modify this proposal to be a "Log Forwarding" service that forwards logs to common/ leading SaaS logging solutions that are used by users who use centralized clouds today (like AWS, Azure, GCP).
2. Identify 2-3 such SaaS logging services (my recommendation is: Datadog, Grafana cloud and Elastic Logstash, in that order of priority) and integrate them to start with and then add more over time (based on customer/ user requests)
3. The integration itself could be one where the user is able to enter auth credentials for their user account in their preferred logging service one time in Console, save it and have it be used for any/ all leases where they wish to forward logs. This can very likely be in the Settings page where the user will select (from a growing list of supported external logging services) a logging service and then enter the required auth info (API Key, Secret etc). Then when the user creates a deployment they will check a box to say "enable logging" - which can be either at deployment creation time or after the deployment is done. In either case, we would need to let the user be able to enable/ disable logging even after the lease is created and ideally would want to let the user be able to do this per lease and not across the board for all leases.

I'll note that the above proposed solution does require that the user be familiar with external logging solutions, how to set them up and how to use them to query logs so this would not be for less technical users (like those who use vercel) - but for now I think this is a good solution to at least address the logging problem for users who understand infrastructure (which I would argue is a vast majority of akash users today). Over time we can look at offering a more seamless experience where the user does not need to configure anything and just pays for a service in Console to "take care of logging" (whether that involves storing logs on akash providers or some other decentralized storage provider like StorJ or Filecoin or to a centralized logging service.

Thanks again for working on this critical problem and we look forward to the updated "Log Forwarding" proposal from you

[HoomanDgtl]

Thank you, @anilmurty and team, for your constructive feedback and suggestions. We will follow up with an updated proposal within this week.

Best regards,
Piyush
Hooman Digital

[pcfreak30]

I am also very interested in this. if this can go to endpoints for granfana's stack self hosted, that would be awesome as well. I would take advantage of this as soon as I could. I am actively R&D'ing some rather complex infrastructure and this will be needed long term.

[HoomanDgtl]

Thank you, @pcfreak30, for taking the time to review our proposal. We will make sure that your request is considered in the final product we’re building.

As Anil mentioned, our initial focus will be on supporting log forwarding to Datadog, Grafana Cloud, and Elastic Logstash, with plans to expand support to additional platforms based on customer and user feedback over time.

[HoomanDgtl]

The proposal is scheduled to go live onchain tomorrow.
thank you

[HoomanDgtl]

The proposal is live onchain: Click here
thank you

[cloud-j-luna]

Hello 👋🏻 thank you for your proposal.
I have a question regarding the implementation. From what I understand this proposal aims to introduce log forwarding at the Console layer. My concern is that I believe perhaps this should be implemented at the protocol level where the feature could be available as primitive in the deployment rather than an injection into the deployment SDL through the console. If this isn't the case, I'd like to see a proposal for the changes in the SDL to allow this to work. Correct me if I'm wrong but as this proposal stands it's only for Console users?

[HoomanDgtl]

Your observation is valid but slightly incorrect. This can be used with all akash deployments, whether they are made with the CLI or the Console. For CLI users, they will need to manually configure the SDL file with the proper setup for our sidecar container. In contrast, Console users benefit from a simplified process with an easy injection through the GUI, requiring only a one-time configuration on the settings page, without any manual typing. However, we will provide documentation and sdl templates to make this feature accessible via the cli. The user would need to fill in the SDL with proper configuration of ENVs, sidecar container and all, but that's pretty much it. On the console, it would require just one click through GUI.

Also you can refer to the POC video here: https://vimeo.com/1033040195/b8f5ee742f?share=copy

[cloud-j-luna]

Thank you for the clarification! I saw the demo and couldn't see a change to the SDL which led me to believe that this was done through a sidecar instantiated by the Console. When you say that you could do it through the CLI you mean that I can create a deployment with a sidecar to export logs, that's a given already without any change to the network. What I mean is a set of primitives to allow exporting without creating a new service in the deployment but rather a setting that I could use to select whether to export logs or not of a container in a deployment. Having this enabled at the protocol level with unlock this feature to more than just Console users out of the box, essentially configuring the log export in the SDL (not directly a sidecar container, but indirectly). Does this make sense?

Also I'd like your opinion @pcfreak30

[HoomanDgtl]

This is primarily a log-forwarding solution designed to send logs from selected deployments to your preferred logging services, such as Datadog, Grafana, and more.

Now, you might wonder—why should a third-party log-forwarding service be implemented into the core protocol? Instead, we’re focusing on building a service that makes it easy to forward logs from Akash to platforms like Datadog, Grafana, etc. This has been a highly requested feature by major clients of Akash, which highlights its value and importance within the Akash ecosystem.

The plan outlined in this proposal has been discussed with all core team members from Akash, as pointed out earlier by @anilmurty. Together, we reviewed the best approach to move forward at this point atleast.

Looking ahead, even if there are future efforts to build this functionality into the protocol itself by core-team, a log-forwarding service will still be essential for forwarding logs from Akash deployments to your desired logging solutions like Datadog, Grafana, and others.

Thanks for your support!

[anilmurty]

hey all - thanks for the comments to this discussion. A few comments/ observations:

1. The goal of this discussion was to find a "client level" solution (as noted in the subject/ headline) to an acute problem that impacts every single customer that deploys production (or close to production-level) services on akash. Can it be implemented at the protocol level? probably but if that is going to take a lot longer to do and if doing a client level solution first does not preclude us from doing a protocol level solution later then I don't think there is a need to prevent this solution fro moving forward
2. When discussions happen on calls and updates are made to existing discussions, there should be notifications to the community to review any changes before posting it on-chain for a formal vote. This can be done 2 ways, I think:

• Post a fresh discussion called "Updated proposal...." and reference the first discussion up top in the new one.
• Or (minimally) update the existing discussion in a way that makes it clear to everyone what has changed and then post a comment asking for feedback on the updates.
  I think this step may have been skipped here which led to people reviewing the discussion for the first time when they saw the on-chain proposal

[cloud-j-luna]

Thank you for the response. I'm not questioning the core team's involvement at all. I'm aligned with the problem being solved.

I understand that solving it in the Console is quicker to deliver value to clients, my points were raised in order to state that a protocol-level solution would enable this to a broader scope of users and integrations, which I and others have particular interest in as we interact mainly through the API.

If possible could you provide supporting documentation (diagrams, RFCs, ...) that support your planned implementation? 😄

[tombeynon]

Hi @HoomanDgtl,

I completely agree that this would be a valuable addition, even as a temporary solution until a protocol-level implementation can be achieved. I have a couple of questions for clarification (apologies for raising these post-proposal):

Open Source Code: Will the resulting code be open-sourced and hosted on the Akash Network GitHub? I didn’t see this explicitly mentioned in the proposal, but I believe it’s an important consideration since the community pool is funding the work. Open-sourcing would not only align with the ethos of the ecosystem but also allow users to extend the solution to other logging providers in the future.

Work Estimate: The estimated 780 hours (~20 weeks) for completion feels quite high at first glance. While I understand this likely reflects the complexity of the task, could you elaborate on the specific challenges that account for this level of effort? For example, does the scope include protocol-level changes to ensure reliable, non-application-specific log access, or are there other factors driving the estimate?

Looking forward to your thoughts and thanks again for pushing this forward!

[HoomanDgtl]

Thank you, @tombeynon, for your feedback.

At Akash, we’ve undertaken significant projects such as the Akash website revamp and enabling deployments on Akash directly from GitHub, live on console. All our work has always remained fully open-source, reflecting our unwavering commitment to open-source principles in everything we build on Akash. Our past contributions are a testament to this dedication and our proven expertise.

When it comes to estimating time and effort, our experience with GitHub-based deployment has made it clear that there is a substantial difference between creating industry and production-ready solutions versus quick fixes to address immediate needs. The proposed work here demands expertise in areas such as frontend and backend development, UI/UX design, shell scripting, DevOps, testing, cloud infrastructure, and more. Additionally, we’ve carefully considered the opportunity cost and long-term vision in crafting this proposal.

This proposal is the culmination of months of deep brainstorming and MVP iterations, ensuring a comprehensive and thoughtful approach.

Thanks again, and I hope this addresses your question.