jasder
/
community
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Add SIG_OHOS_Kernel.md & SIG_OTA_Updates.md & SIG_Tools_Toolchains.md

This commit is contained in:
L97229 2021-02-08 00:33:30 +08:00 committed by Gitee
parent fd2c34705b
commit d755d35422
3 changed files with 319 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

72
sig/SIG_OHOS_KERNRL.md Normal file
View File

@ -0,0 +1,72 @@
# Open Source Kernels in OpenHarmony OS
[TOC]
## 1. Summary
Kernels are the SW abstraction that allow us to run applications on a broad range of HW. By themselves, kernels are not very interesting since they merely act on the behalf of applications to access HW resources. However, it is important that kernels are extremely resource efficient and secure so as to prevent hostile code from taking over HW.
From the point of view of OpenHarmony, the goal is to ensure we support the best-in-class kernels required to run our applications as efficiently as possible to as many HW platforms as possible in a secure manner.
### 1.1 Goals
The role of this core group is to ensure that OpenHarmony supports the broadest range of CPUs, SoCs, peripherals and communication technologies - via buses or network - with open-source kernels. This SIG will work on identifying key technologies that need to be integrated into the kernels to provide a tightly integrated and performant HW abstraction to run OpenHarmony Services and applications on.
### 1.2 Scope
This group will have the following high-level goals:
1. Continuously evaluate support for best-in-class kernels to address broadest possible SoC and peripheral support
1. Evaluate new kernels that might be proposed by partners for integration into OpenHarmony in terms of what value it brings to the Ecosystem
1. Work with other core groups and partners to identify key technologies that need kernel integration to enable new usecases and bring it into OpenHarmony
1. Work with the Application Framework teams to provide an abstraction layer for each technology that needs to be exposed to Application developers independent of the underlying kernel
1. Contribute to and influence upstream projects on technical issues with integrating upstream kernel components into solutions
1. Work with SoC and chipset vendors to provide support for their HW upstream and in OpenHarmony
1. Maintain a list of supported peripherals for each vertical solution to enhance cooperation and utilization within the OpenHarmony ecosystem
### 1.3 Repository
### 1.4 Non-Goals
The following are not goals of this group:
1. Writing new kernels: While the group will review proposals to include support for other kernels and other tightly integrated components such as network stacks, the main focus will be integration of the components on technical merit and value it brings to the ecosystem. Writing new kernels is not a goal.
1. Writing new drivers: The group will work with SoC and chipset vendors to integrate their drivers into OpenHarmony, not write them from scratch.
### 1.5 Proposal
The kernel core group will be responsible for maintaining the kernel support and architecture for OpenHarmony. Since the kernel talks to all the hardware, it is expected that this group will be involved in discussions in every part of the vertical solution stack e.g. multimedia, display subsystem, security, network technologies.
The group will ensure that any external component being considered for integration into OpenHarmony that requires close integration with the kernel will work across all the supported kernels.
#### 1.5.1 Story 1
OpenHarmony wants to add support for some network technology. The kernel core group will ensure that the exposed networking abstraction layer allows Linux and RTOS versions of the network technology to exist and work efficiently with the respective kernel network stacks.
#### 1.5.2 Story 2
A partner wants to add support for a new RTOS to OpenHarmony. The kernel core group will evaluate the value this RTOS will bring to the ecosystem, the cost of porting the application layer to this new RTOS and supporting it inside OpenHamony. The group will decide on the what test suites are applicable for the new RTOS and monitor the results before support is added to OpenHarmony.
#### 1.5.3 Story 3
Security team wants to introduce new techniques to secure OpenHarmony. The kernel core group will work with the security team to integrate these features as transparently as possible
#### 1.5.4 Story 4
Upstream community needs help test and validate the stability of release candidates for the next LTS kernel. The kernel core group will help provide feeedback to upstream maintainers by the way our CI and test reports.
#### 1.5.5 Story 5
Openharmony needs to decide on the supported kernel versions for the next release. This decision needs to take into account support for reference hardware from the partners, each kernel's LTS schedules and feature roadmaps.
## 2. SIG Related information
### 2.1 Maintainers
* Amit Kucheria [@idlethread]
### 2.2 Mailing Lists
* would be nice to create a new ML for kernel. kernel@openharmony.io?
### 2.3 Slack group
We use an Open Source alternative to Slack called Mattermost.
The kernel core group inside the OSTC Mattermost instance, hosted in EU, lives at [https://chat.ostc-eu.org/ostc/channels/kernel](https://chat.ostc-eu.org/ostc/channels/kernel).

164
sig/SIG_OTA_Updates.md Normal file
View File

@ -0,0 +1,164 @@
# Over The Air (OTA) Updates
## SIG work objectives and scope
### Summary
Over The Air (OTA) updates is one of the core functionalities of products aiming for post-release support and maintenance. In the context of OpenHarmony OS, the update mechanism (or mechanisms) need be kernel agnostic (for the supported types of kernels) from the perspective of the user. We define here a “user” as being the operator of the device deployment and roll-out activities. Also, the same mechanisms need to provide a seamless experience to roll-out updates targeting different types of entities: a device, a fleet, a group, a fleet filter etc.
The OTA is one of the main functionalities that cross the software stack from the BSP to the OS, to the update client and up to the server endpoint. The high importance of this topic is driven by the fact that the project needs to have a robust post-initial deployment path for maintaining devices, fixing bugs in the OS, updating the OS with new functionality and deploying security fixes.
The rationale behind this core component is, at its core, the acknowledgement that software evolves. This evolution can happen due to various factors: e.g. a regression in a certain functionality, improved or new functionality, security fixes. These evolutions are easily handled when the device is readily available, on one's desk, provided that there is a flashing (loading, deploying) path for that specific device. The OTA comes to fill the gap where the targeted device, or fleet of devices, cross the boundaries of physical access and rely heavily on network connectivity. Devices in such an environment, need to run update software that can’t rely on stable power and network access while still being robust and consistent with its operations.
It is worth emphasizing the fact that while software updates target various use cases, regular security fixes are standard for maintaining trust with the end-users. That is especially valid when the software stack is based on a variety of opensource components that in turn are tracked and provide security reports and fixes regularly. All these security updates need to be surfaced by the OpenHarmony OS software stack through its update mechanism. As an example, Android maintains a monthly security bulletin since August 2015 providing security fixes for its disclosed vulnerabilities.
### Work goals
The main goal for a successful OTA system is to be able to deploy software on devices in an “atomic”, safe and robust way. This responsible team will focus on designing such a system, taking advantage of the extensive experience and expertise of the team.
Before addressing the update story and goals, the user experience starts with the board software loading or flashing. To satisfy this, there needs to be a clear and unified process to load the OpenHarmony OS software stack to a target in one of the following states:
* no software is loaded on the target
* OpenHarmonyOS incompatible software is running on the target
These software loading processes will assume physical access.
All software on the device needs to be updateable – subject to the manufacturer’s exposed capabilities and also subject to the “update” design. For example, in an A/B system, everything needs to be updatable up the point of where the indirection happens. Beyond that point, the update can happen but under different operation requirements (for the accessible software blobs).
The OTA system must hide the target’s hardware and software (kernel) specifics when providing the update process. From the perspective of deploying an update, doing that on target type A, or target type B should not affect the update experience (where A and B are any supported targets).
When handling update operations, the system must be able to target individual entities, groups of entities (fleets) and be also able to extend this to filtered groups.
The system will provide device telemetry data.
The update process needs to be modelled with a concept of “update strategy”. This would allow a configuration of how the process is handled when deploying an update to multiple targets. These strategies will consider restricting factors (e.g. bandwidth, location).
Minimizing the transferred data when performing update actions is to be considered core functionality since IoT devices are usually under metered or pay-as-you-go data transfer channels.
The device support will provide a clear process allowing new targets to easily be adapted to the update platform.
The update system design will prioritize reusing existing opensource solutions that are in line with the system’s requirements. When these components are identified, the team will approach the development process with an “upstreaming-first” mindset. Forks of opensource components are to be treated as staging areas while the upstream process takes place.
Security is another important goal of this update system and aims at designing a system that relies on proven security practices, leveraging existing cryptography solutions. In doing so, the strategy of storying, deploying and consuming update payloads needs to assume all the aspects of the security requirements a per the team’s considerations.
#### Non-Goals
The OTA software stack is not aiming for 100% ability to update the device software stack due to eventual hardware limitations and robustness requirements.
Manual modifications on remote targets are not in the scope of this document.
Device OS metamorphosis (migrating an OS without support for the HarmonyOS update system to one that does) is to be treated separately if at all.
The team will minimize internal development while maximizing opensource components usage and contributions. With that as a goal, it means that a non-goal for this document is to develop an entire update system stack from scratch.
The OTA update system is only responsible through its design and implementation of the potential attacks and vulnerabilities to the update process itself. It doesn’t protect or is concerned about the security vulnerabilities outside of the update process. For this kind of attacks, the OS needs to provide its security components and strategies. Even so, the update system will facilitate, provide a channel for addressing OS security concerns/issues by being the updating "vehicle".
#### Design details
The update system is to be separated into three main components:
1. A server stack. This is the service provider of the update system including a management dashboard. It adheres to the redundancy and availability requirements under the defined cloud policy.
2. A device update client. This is a software component (or a set of software components) in charge of querying for an update, consuming the update and providing metrics to the backend.
3. The OS. The OS needs to provide a foundation for the update stack to ensure that the update system requirements are satisfied. This can mean stateless root filesystem, atomic bootloader switch etc.
There are three major challenges in designing this system.
The system must be able to provide a fallback and act accordingly based on update health checking.
The second challenge is about communication. The update payload must be sent to the device so the protocol must be accounted for in the software design.
The last challenge is security. The targets must be able to verify the update payload proving in this way that the server is a trusted party (authentication). While the payload is transferred, its content must be obfuscated as it might contain sensitive information (confidentiality). Also, the update payload has to provide means of checking its integrity on the client-side. This will allow detecting corrupted payloads transmitted over the network (integrity).
The key elements of an update system are:
* Power-cut and network instability resilience
* Scalability
* Security
IoT devices are usually deployed in environments where network and power are not assumed to be stable. The software stack needs to be able to cope with unexpected power cuts (and/or network access). To handle this, updates need to be atomic and durable. That also applies for any filesystem writes (at the level of filesystem entries) but it is vital for robust updates as a transaction.
Once there is a robust update system available, and a process to use it, the next important aspect is scalability. The system needs to handle an individual entity (one target) using the same update process as with a fleet entity.
Security is one of the core activities of an update system. In the scope of the update system, there are three main areas of concern:
* Server-side
* Client-side
* Tools
One the server-side, there are decisions to be taken in terms of where to host the instance(s), how to design and implement the communication in between the client (a target) and the server, user authentication and management, how to authenticate the devices and uniquely identify a target along with designing and implementing a logging and reporting mechanism for the update framework. Part of these decisions will also affect or shape the process of storing the update payloads and the key management of the target devices.
On the client-side, the communication to the server is one of the important aspects. It should be encrypted and provide the ability of the target to verify that the communication happens with the expected server endpoint while guarding against man-in-the-middle attacks. The way the devices store their keys is also an important aspect. This is because these devices are to be assumed in hostile environments therefore using hardware-based security keys is preferable. The client should also be able to verify the update payload as authentic and unmodified - digital signatures.
Lastly, on the tooling-side of security, the update system needs to provide primitives and processes to replace keys over time. Also, the systems processing the payloads and signing updates will need to be carefully designed while still being able to produce the required artefacts. Tools are to be selected to manage these secrets at the level of CI, build system and server endpoint deployment.
All these concerns are the primary focus of this update system.
With these principles, concerns and guidelines in mind, we can move on to defining update OS strategies.
The first update strategy, usually also the standard in the Linux-based embedded systems, is the dual copy update (A/B). It implies setting up an inactive partition before atomically switching the bootloader to use it at next boot. The main disadvantage is the fact that it doubles the OS disk usage and also, the OS and the application are treated as one update payload. The advantage is that we can update the entire OS in one go, as an atomic operation. This strategy can easily support implementing fallback mechanisms as the current inactive is usually the old OS (so a health fail detection on active can always fallback). On top of this strategy, we can use existing components like RAUC and SWUpdate along with existing management systems like HawkBit. Mender is also another solution but the management update server is not HawkBit compatible. using this strategy, the atomicity will be at the level of the root filesystem - image-based updates. The update can also treat as a transaction multiple filesystem updates in more complex scenarios.
The second strategy is container-based updates. The main idea here is that there is a separation in between apps and OS where the OS provides the means to run and manage the lifecycle of the application containers through an agent which in turn is managed by a cloud-based management system. Balena.io is a full-featured solution in this segment where each application is packaged as a Docker image, updates are deployed as Docker images which trigger the agent to respawn local containers. There is extra complexity on how to deploy a stack of components with dependencies and various extra deployment strategies but the main idea is that using this kind of system, there is a clear separation in between applications and OS in terms of updates deployment. There is extra care needed here because the container engine needs to also adhere to the robustness principles outlined above (eg. power-cut resilience). This strategy raises the transaction at the level of the container/application image guarding the atomic operation as the application container.
In the same container-based strategy, various other tools can be used to maintain and deploy updates – for example, “osTree” for delta generations and “runc” for running the containers. This can be combined to generate a Balena similar system without the complexity of a container engine like “Docker”.
The simplest strategy and the last one raised in this document (only mentioned for completeness) is a single copy update. This strategy handles the update while the system is unusable and also, it is hard to implement power failure resilience because the update partition is also the same one used to run the OS.
The strategies above are mainly targeting Linux-based system. This is because components like containers are not feasible for “edge” devices. Both ZephyrOS and FreeRTOS have OTA capabilities (implemented through various primitives) but to make the update experience simple, no matter the device type/kernel, the proposal is to consider a Linux gateway. Such a gateway device would act as an updater application, that is aware (can detect) of it’s connected MCUs and knows the primitives to act upon deploying updates.
On the side of the update payloads, the content will be defined in terms of its scope with implications on the risk and security aspects. A payload can be classified as:
* patch update: security fix, CVE fix, no impact refactoring etc.
* minor, major: updates of core components that break currently assumed ABI/API (compiler, kernel, etc) as opposed to "leaf updates" that only deal with downstream components where few dependent components might be affected.
These payload classifications can later be used as filters or "update rules" for managing risk and frequency/size requirements.
Update payloads will be minimized (or optimized) for speed and size. Both aspects are closely related but they target two different perspectives: speed provides a UX feature while size provides a functional feature where devices are restricted in terms of data transfer (e.g. targets over 5G network). There are various strategies in this regard:
* Use minimum deltas of only the strictly changed blobs in the payload (it will always be specific to the combination of running version and target version making it the heaviest on the delta generating side).
* Use optimized deltas where they provide changed blobs for a set of versions (easier on the delta generating server but heavier in size).
* Use update strategy restrictions at a per target or per group level. This means that the model should have a concept of "update rules" where an entity (target, group, fleet) can be enriched with metadata of types updates the entity accepts. A simple example is having a fleet that only accepts major updates while having another fleet accepting only security fixes. This approach has implications on the risk management side as well. A fleet "listening" only on the "major updates" channel will take less risk as opposed to one listening to any update provided by the update server endpoint.
The loading/flashing strategy is to be defined at the target level. For Linux based system there are proven strategies used by various existing solutions in production while keeping the approach as generic as possible. For example, once a bootloader can boot from an external storage device (or NFS), a flasher “initramfs” routine packaged with a target OS image, can proceed in loading the OS, providing visual progress when available and rebooting into the flashed system. For other targets, manufacturing tools are to be used when available – for example, “mfgtools”.
The implementation details should provide a stateless flashing mechanism such that the same flashing payload can be used on multiple targets. For example, a USB stick loaded with a “flasher” image, would not maintain state in between flashing processed so that each operation would produce the same outcome.
The stateless aspect is also important when designing the OS to facilitate various functionalities of the update system. A design separating the OS and the state of the device will make the update process simpler. This can be done using separate partitions leveraging symlinks, bind mounts or an overlay structure. This also enables consuming update deltas as opposed to always downloading a full software stack. There are more implementation details to be considered here: version-specific deltas as opposed to designing deltas applicable to multiple versions from the same baseline. These strategies and solutions are to be carefully considered by the responsible team.
### Scope of work
* Identify OTA requirements
* Itentify OTA strategies to be acting as solution umbrella
* Formalise OTA architecture
* Decide on opensource tools and projects to use in implementation
* Provide support and expertise for applications to adhere to update principles - eg. atomicity, durability
* Decide and evaluate frontend solutions
* Define update flow for both production and development cycles
Once the decisions and strategies are defined, the OTA modules will be maintained and built by the SIG team.
The OTA will have an upstream-first mindset to send changes to the relevant upstream projects in a timely manner. The local changes will be kept as a staging work, temporary. Depending on the actual case, that can happen in git (as branches) or part of the upstream project architecture (for example build system _layers_ in Yocto Project).
### Repository names and descriptions
* planning
* Design documents keeping track of why some decisions were made, issue tracker to help with project management
* docs
* Documentation for users of the tools and toolchains
* components
* Group of repositories.
* Software components part of the OTA stack.
Further repositories may be created in the future based on the SIG's needs.
## SIG Related information
### Maintainers
* Andrei Gherzan [@agherzan] <andrei.gherzan@huawei.com>
### Mailing Lists
It would be nice to create a new ML for this SIG. _ota@openharmony.io_ can be an option.
### Slack group
We do not use Slack. In its place, we use an Open Source alternative to Slack called Mattermost. The OTA group inside the OSTC Mattermost instance, hosted in a country not known for imposing bans on Huawei, lives at [https://chat.ostc-eu.org/ostc/channels/toolchain](https://chat.ostc-eu.org/ostc/channels/ota).

83
sig/SIG_Tools_Toolchains.md Normal file
View File

@ -0,0 +1,83 @@
# Tools and toolchain SIG
## SIG work objectives and scope
### Summary
Tools and toolchains are an important part of building any operating system.
They have a big impact on performance, compatibility and developer
friendliness.
It is therefore important for OpenHarmony to make the right decisions about
tools and toolchains, making a SIG necessary.
### Work Goals
1. Determine the best tool and toolchain options currently available
2. Optimize the chosen components for OpenHarmony - set reasonable defaults with OpenHarmony triplets, make sure OpenHarmony is a visible target in IDEs, etc.
3. Push modifications to toolchain projects to LLVM and gcc upstream - any local patches to toolchain components should be temporary. This includes helping other teams like (non-Open) Harmony upstream their changes if any/if needed.
4. Keep monitoring the Open Source world for new versions (the toolchain should track upstream releases - they usually come with better performance and compliance with newer language standards) and entirely new projects (including, but not limited to, Huawei's OpenArk compilers) that could improve OpenHarmony tools and toolchains, update OpenHarmony tools and toolchains in a timely manner
5. Provide expertise/assistance to developers running into toolchain problems (such as compiler bugs, or incompatible changes in newer releases caused by features like stricter standards compliance)
6. Make sure people trying to build OpenHarmony and/or develop OpenHarmony apps can easily install the needed toolchain components regardless of their work environment
7. Identify and adapt debugging, profiling and code quality tools and sanitizers that can be useful for the OpenHarmony ecosystem. If a new tool needs to be developed, try to develop it inside an upstream project to bring in external developers/maintainers.
8. Decide on components and architecture of the OpenHarmony SDK for hardware makers and application/game developers
9. Avoid host tool contamination - help make sure that, when building the OS, the OpenHarmony toolchain is used instead of potentially equivalent host tools that may come with a different set of bugs
10. Help OpenHarmony developers with toolchain problems
11. Make sure OpenHarmony is friendly to developers regardless of whether they choose to use an IDE or a more traditional environment (e.g. vim/emacs and command line tools)
#### Non-Goals
1. Writing new toolchain components from scratch unless absolutely needed
2. Forking toolchain components (if forks can't be avoided altogether [e.g. to get (Open)Harmony support before upstream projects start to care], any forks should be temporary until patches are in a shape that is acceptable upstream).
#### Design Details
While investigating a tool, the Tools and Toolchain SIG has to keep the following criteria in mind:
1. Tool must be available under a compatible Open Source license.
2. Tool should have a clear future
3. Tool should deliver the best performance among comparable tools (or be significantly better in code quality and/or development speed, to the point that we can be reasonably sure it will match/outperform comparable tools in the not too distant future)
4. Tool should not be limited to a particular host system
5. Where possible, tools that can be shared across different target systems (hardware architecture as well as kernel choice) should be preferred over tools that introduce unnecessary differences. While some toolchain components may need different builds for Linux based OpenHarmony, Zephyr based OpenHarmony, LiteOS based OpenHarmony and FreeRTOS based OpenHarmony, they should be built from the same source to the extent possible.
6. While it likely makes sense to provide an easy to use IDE, a developer's choice should not be limited to that IDE. Developers using an IDE should be supported just as well as developers using more traditional tools like text mode editors and commands.
7. Where at all possible, situations like "Board X needs an old kernel that can be built only with an ancient toolchain, so we use a separate kernel compiler and userland compiler" should be avoided - preferably by keeping kernel support for relevant devices up to date, if necessary by backporting patches to support newer toolchains to kernels that are still relevant.
8. Code reuse should be maximized, and porting/rework efforts should be minimized.
### Scope of work
The Tools and Toolchain SIG looks into the following questions (more will be added later as OpenHarmony progresses):
1. gcc or clang?
2. libgcc or compiler-rt?
3. libstdc++ or libc++?
4. Do we want to include (optional) support for additional programming languages, such as Java, C#, Go, Rust, Swift, Python or Lua? If so, which languages and which implementations thereof? Which runtimes (if any) go into the core OS, which are provided as optional components inside the SDK?
5. libc: Currently Linux and LiteOS-A use Musl, Zephyr uses a newlib fork, FreeRTOS is typically used with newlib. There are good reasons to use different libcs here (newlib doesn't support dynamic linking, which is very important for higher end systems, while musl doesn't support MMU-less systems), but there may be some things we can do to improve compatibility between OpenHarmony on top of different kernels, and some ways to optimize both worlds while reducing maintenance work (e.g. unify the implementations of some functions in both libcs, or even bring in more optimized versions of some routines from other libc implementations like glibc or Android's Bionic)
6. Which debugging tools, profilers, code quality tools, sanitizers etc. do we need/want on OpenHarmony?
7. What IDE or IDEs, if any, do we want to modify for improved OpenHarmony support? How can we best support IDE users and traditional tool users alike?
8. Do we assemble the toolchain (not its components; that is beyond the scope) from scratch or do we base our work on an existing project (e.g. OpenEmbedded/Yocto, Buildroot, a Linux distribution's packaging, ...)
Once decisions are made, the SIG builds and maintains the tools and toolchains used in OpenHarmony and the SDK (tools, library headers, etc. and optional IDE(s) as separate modules).
The toolchain is provided in binary form and in source form with a simple build script that includes all configurations etc. used for the official build.
If a tool bug is reported to the SIG, it responds by fixing the bug or passing the bug report on to a relevant upstream project with all needed information (possibly adding details like a fully reduced test case for a compiler bug to the original report), and by providing a workaround if the problem can't be fixed properly in a timely manner.
Patches for bug fixes and optimizations should always go upstream in a timely manner. Local patches to upstream components should be temporary at most (and kept in git branches that can be rebased easily). This minimizes overhead and maximizes reuse.
### Repository names and descriptions
* planning Design documents keeping track of why some decisions were made, issue tracker to help with project management
* docs Documentation for users of the tools and toolchains
* packaging Scripts and helpers for generating installable packages of the toolchain components for various operating systems/distributions
* llvm-project A mirror of [LLVM's git repository](https://github.com/llvm/llvm-project.git) with additional branches containing modifications not yet upstreamed
Further repositories may be created in the future.
## SIG Related information
### Maintainers
* Bernhard "bero" Rosenkraenzer [@berolinux]
### Mailing Lists
* would be nice to create a new ML for tools and toolchains. toolchain@openharmony.io?
### Slack group
We do not use Slack. In its place, we use an Open Source alternative to Slack called Mattermost.
The toolchain group inside the OSTC Mattermost instance, hosted in a country not known for imposing bans on Huawei, lives at [https://chat.ostc-eu.org/ostc/channels/toolchain](https://chat.ostc-eu.org/ostc/channels/toolchain).