[Feature][Task] Support config max task log size for per task

[ruanwenjun]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Avoid large log file appear.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[dill21yu]

Currently, DolphinScheduler uses FileAppender for task logs without size limits, which can lead to large log files that consume excessive disk space.
Is this a feasible solution?
Replace FileAppender with RollingFileAppender and implement size-based rolling policy for task logs, similar to existing MASTERLOGFILE and WORKERLOGFILE configurations.

1. Configuration Changes:
   dolphinscheduler-master/src/main/resources/logback-spring.xml
   dolphinscheduler-worker/src/main/resources/logback-spring.xml
   dolphinscheduler-standalone-server/src/main/resources/logback-spring.xml
   Replace the TASKLOGFILE appender:
   <appender name="FILE-${taskInstanceLogFullPath}" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${taskInstanceLogFullPath}</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <fileNamePattern>${taskInstanceLogFullPath}.%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>${task.log.max.file.size}</maxFileSize> <maxHistory>${task.log.max.history}</maxHistory> <totalSizeCap>${task.log.total.size.cap}</totalSizeCap> <cleanHistoryOnStart>true</cleanHistoryOnStart> </rollingPolicy> <encoder> <pattern>%date{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%thread] - %message%n</pattern> <charset>UTF-8</charset> </encoder> <append>true</append> </appender>

Add configurable properties:
<property name="task.log.max.file.size" value="200MB"/> <property name="task.log.max.history" value="3"/> <property name="task.log.total.size.cap" value="600MB"/>
Taking max.file.size=1MB as an example

2. Log Query Logic Updates
   Modify logClientDelegate.getPartLogString() to handle multiple rolled files
   Implement file discovery logic to find all rolled log files
   Add cross-file line number calculation and content aggregation
   Update log download functionality to aggregate all rolled files

And I am willing to submit a PR! @ruanwenjun @SbloodyS

[ruanwenjun]

@dill21yu LGTM, but can it works? I am not sure if the usage is correct in logback.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.