【黑马头条】-day04自媒体文章审核-阿里云接口-敏感词分析DFA-图像识别OCR-异步调用MQ

---

文章目录 day4学习内容自媒体文章自动审核今日内容 1 自媒体文章自动审核1.1 审核流程1.2 内容安全第三方接口1.3 引入阿里云内容安全接口1.3.1 添加依赖1.3.2 导入aliyun模块1.3.3 注入Bean测试 2 app端文章保存接口2.1 表结构说明2.2 分布式id2.2.1 分布式id-技术选型2.2.2 雪花算法2.2.3 配置雪花算法 2.3 保存app端文章-思路分析2.4 实现接口2.4.1 实现步骤2.4.2 定义feign接口2.4.2.1 导入feign远程调用依赖2.4.2.2 定义文章端远程接口2.4.2.3 导入ArticleDto 2.4.3 实现feign接口2.4.4 创建mapper2.4.5 为AparticleConfig设置默认参数2.4.6 在ApArticleService的实现类ApArticleServiceImpl中实现方法2.4.7 启动ArticleApplication 3 自媒体文章审核实现3.1 创建审核接口3.2 实现审核接口3.3 启动类扫描feign3.4 测试 4 自媒体调用文章微服务feign远程调用服务降级4.1 feign远程调用服务降级处理的逻辑4.2 编写降级逻辑4.3 指定IArticleClient接口指向Feign降级逻辑4.4 加载feign降级逻辑4.5 配置降级策略4.6 测试 5 文章审核异步调用5.1 在自动审核的方法加上@Async注解5.2 在文章发布后调用自动审核方法5.3 在启动类中添加注解开启异步调用5.4 综合测试5.5 使用rabbit MQ来完成异步调用5.5.1 引入依赖5.5.2 为微服务配置MQ5.5.3 改造方法，创建监听队列5.5.4 序列化MQ消息5.5.5 加上mq后的综合测试 6 自管理敏感词过滤6.1 DFA实现原理6.2 DFA检索过程6.3 实现步骤6.3.1 创建敏感词表6.3.2 将wm_sensitive对应的实体类和mapper导入6.3.3 在阿里云接口前自行进行审查6.3.4 测试 7 图片文字敏感词过滤7.1 文字图片识别7.2 Tesseract-OCR7.3 Tess4j案例7.3.1 导入依赖7.3.2 将训练好的分类器放入资源中7.3.3 demo7.3.4 结果 7.4 图片文字敏感词过滤实现7.4.1 创建工具类7.4.2 工具类被其他微服务使用7.4.3 在微服务中配置7.4.4 添加实现 8 静态文件生成8.1 实现思路8.1.1 生成minio接口和实现，并且异步调用8.1.2 修改saveArticle逻辑8.1.3 开启异步调用8.1.4 测试

---

day4学习内容 自媒体文章自动审核

今日内容

1 自媒体文章自动审核 1.1 审核流程

1.2 内容安全第三方接口

1.3 引入阿里云内容安全接口

1.3.1 添加依赖

在heima-leadnews-common包下引入依赖

<dependency> <groupId>com.aliyun</groupId> <artifactId>aliyun-java-sdk-core</artifactId> <version>4.1.1</version> </dependency> <dependency> <groupId>com.aliyun</groupId> <artifactId>aliyun-java-sdk-green</artifactId> <version>3.6.6</version> </dependency> <dependency> <groupId>com.alibaba.fastjson2</groupId> <artifactId>fastjson2</artifactId> <version>2.0.9</version> </dependency> <dependency> <groupId>com.aliyun.oss</groupId> <artifactId>aliyun-sdk-oss</artifactId> <version>2.8.3</version> </dependency> 1.3.2 导入aliyun模块

放入heima-leadnews-common模块下的com.heima.common

哪个微服务使用，就在哪个微服务的nacos中配置

在heima-leadnews-wemedia中的nacos配置中心添加以下配置：

aliyun: accessKeyId: LTAI5tCWHCcfvqQzu8k2oKmX secret: auoKUFsghimbfVQHpy7gtRyBkoR4vc #aliyun.scenes=porn,terrorism,ad,qrcode,live,logo scenes: terrorism 1.3.3 注入Bean测试

在resource中META-INF的spring-factories中自动配置

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\ com.heima.common.exception.ExceptionCatch,\ com.heima.common.aliyun.GreenTextScan,\ com.heima.common.aliyun.GreenImageScan

在测试类中进行测试

@SpringBootTest(classes = WemediaApplication.class) @RunWith(SpringRunner.class) public class AliyunTest { @Autowired private GreenTextScan greenTextScan; @Autowired private GreenImageScan greenImageScan; @Autowired private FileStorageService fileStorageService; @Test public void testScanText() throws Exception { Map map = greenTextScan.greeTextScan("我是一个好人,冰毒"); System.out.println(map); } @Test public void testScanImage() throws Exception { byte[] bytes = fileStorageService.downLoadFile("http://192.168.200.130:9000/leadnews/2021/04/26/ef3cbe458db249f7bd6fb4339e593e55.jpg"); Map map = greenImageScan.imageScan(Arrays.asList(bytes)); System.out.println(map); } } 2 app端文章保存接口 2.1 表结构说明

2.2 分布式id

2.2.1 分布式id-技术选型

2.2.2 雪花算法

2.2.3 配置雪花算法

第一：在实体类中的id上加入如下配置，指定类型为id_worker

@TableId(value = "id",type = IdType.ID_WORKER) private Long id;

第二：在application.yml文件中配置数据中心id和机器id

在文章的微服务的nacos配置中leadnews-article中添加

spring: datasource: driver-class-name: com.mysql.jdbc.Driver url: jdbc:mysql://localhost:3306/leadnews_article?useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC&useSSL=false username: root password: 123sjbsjb # 设置Mapper接口所对应的XML文件位置，如果你在Mapper接口中有自定义方法，需要进行该配置 mybatis-plus: mapper-locations: classpath*:mapper/*.xml # 设置别名包扫描路径，通过该属性可以给包中的类注册别名 type-aliases-package: com.heima.model.article.pojos #雪花算法 global-config: datacenter-id: 1 workerId: 1 minio: accessKey: minio secretKey: minio123 bucket: leadnews endpoint: http://192.168.204.129:9000 readPath: http://192.168.204.129:9000

2.3 保存app端文章-思路分析

2.4 实现接口

2.4.1 实现步骤

2.4.2 定义feign接口 2.4.2.1 导入feign远程调用依赖

在heima-leadnews-feign-api的pom.xml中导入依赖

<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-openfeign</artifactId> </dependency> 2.4.2.2 定义文章端远程接口

heima-leadnews-feign-api定义com.heima.apis.article.IArticleClient接口

@FeignClient(value = "leadnews-article")

@FeignClient指定文章远程调用接口名称

@FeignClient(value = "leadnews-article") public interface IArticleClient { @PostMapping("/api/v1/article/save") public ResponseResult saveArticle(@RequestBody ArticleDto dto) ; } 2.4.2.3 导入ArticleDto

在heima-leadnews-model模块下com.heima.model.article.dto中导入ArticleDto类

@Data public class ArticleDto extends ApArticle { /** * 文章内容 */ private String content; } 2.4.3 实现feign接口

在heima-leadnews-service模块下的heima-leadnews-article模块下创建com.heima.article.feign.ArticleClient类

@RestController public class ArticleClient implements IArticleClient { @Autowired private ApArticleService apArticleService; @PostMapping("/api/v1/article/save") @Override public ResponseResult saveArticle(@RequestBody ArticleDto dto) { return apArticleService.saveArticle(dto); } } 2.4.4 创建mapper

在heima-leadnews-service模块下的heima-leadnews-article模块下创建com.heima.article.mapper.ApArticleConfigMapper接口

@Mapper public interface ApArticleConfigMapper extends BaseMapper<ApArticleConfig> { } 2.4.5 为AparticleConfig设置默认参数

添加@NoArgsConstructor

public ApArticleConfig(Long articleId) { this.articleId = articleId; this.isDelete = false; this.isDown = false; this.isForward = true; this.isComment = true; }

添加有参构造

2.4.6 在ApArticleService的实现类ApArticleServiceImpl中实现方法

ApArticleService接口

public interface ApArticleService extends IService<ApArticle>{ /** * 加载文章列表 * @param dto * @param type 1 加载更多 2 加载最新 * @return */ public ResponseResult load(ArticleHomeDto dto, Short type); /** * 保存文章 * @param dto * @return */ public ResponseResult saveArticle(ArticleHomeDto dto); }

实现类，实现saveArticle方法

@Autowired private ApArticleConfigMapper apArticleConfigMapper; @Autowired private ApArticleContentMapper apArticleContentMapper; /** * 保存文章 * @param dto * @return */ @Override public ResponseResult saveArticle(ArticleDto dto) { //1.参数检查 if(dto == null){ return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID); } ApArticle apArticle = new ApArticle(); //org.springframework.beans BeanUtils.copyProperties(dto, apArticle); //2.判断是否存在id if(dto.getId() == null) { //2.1 不存在id ，新增 文章、内容、配置 save(apArticle); //2.1.2 保存文章配置 ApArticleConfig apArticleConfig = new ApArticleConfig(apArticle.getId()); apArticleConfigMapper.insert(apArticleConfig); //2.1.3 保存文章内容 ApArticleContent apArticleContent = new ApArticleContent(); apArticleContent.setArticleId(apArticle.getId()); apArticleContent.setContent(dto.getContent()); apArticleContentMapper.insert(apArticleContent); } else { //2.2 存在id，更新 文章、内容 //2.2.1 更新文章 updateById(apArticle); //2.2.2 更新文章内容 ApArticleContent apArticleContent = apArticleContentMapper.selectOne(Wrappers .<ApArticleContent>lambdaQuery() .eq(ApArticleContent::getArticleId, dto.getId())); apArticleContent.setContent(dto.getContent()); apArticleContentMapper.updateById(apArticleContent); } //3.返回结果 文章的id return ResponseResult.okResult(apArticle.getId()); } 2.4.7 启动ArticleApplication

刚刚是新增，如果是修改。

就会在json中传入id

成功修改

3 自媒体文章审核实现 3.1 创建审核接口

在heima-leadnews-service中heima-leadnews-wemedia中的service新增WmNewAutoScanService接口

public interface WmNewAutoScanService { /** * 自动审核媒体文章 */ public void autoScanMediaNews(Integer id); } 3.2 实现审核接口 @Service @Slf4j @Transactional public class WmNewAutoScanServiceImpl implements WmNewAutoScanService { @Autowired private WmNewsMapper wmNewsMapper; @Qualifier("com.heima.apis.article.IArticleClient") @Autowired private IArticleClient iArticleClient; @Autowired private WmChannelMapper wmChannelMapper; @Autowired private WmUserMapper wmUserMapper; @Override public void autoScanMediaNews(Integer id) { //1.查询自媒体文章 WmNews wmNews = wmNewsMapper.selectById(id); if (wmNews == null) { throw new RuntimeException("自媒体文章不存在"); } if(wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())){ Map<String,List<String>> scanMaterialsList = extractImageAndContent(wmNews); //2.调用阿里云接口审核文本内容 List<String> contentTexts =scanMaterialsList.get("contentTexts"); boolean isTextScan =true; if(!isTextScan)return; //3.调用阿里云接口审核图片内容 List<String> imagesUrls =scanMaterialsList.get("imagesUrls"); boolean isImageScan =true; if(!isImageScan)return; if(isTextScan && isImageScan) { //审核通过 wmNews.setStatus((short) 9); wmNews.setReason("审核通过"); } } //4.审核成功保存app端的相关文章数据 ArticleDto dto=new ArticleDto(); BeanUtils.copyProperties(wmNews,dto); //布局 dto.setLayout(wmNews.getType()); //频道 dto.setChannelId(wmNews.getChannelId()); //频道名称 WmChannel wmChannel = wmChannelMapper.selectById(wmNews.getChannelId()); if(wmChannel!=null){ dto.setChannelName(wmChannel.getName()); } //作者 dto.setAuthorId(Long.valueOf(wmNews.getUserId())); //作者名称 WmUser wmUser= wmUserMapper.selectById(wmNews.getUserId()); if(wmUser!=null){ dto.setAuthorName(wmUser.getName()); } //设置文章id if(wmNews.getArticleId()!=null){ dto.setId(wmNews.getArticleId()); } dto.setCreatedTime(new Date()); ResponseResult responseResult = iArticleClient.saveArticle(dto); if(responseResult.getCode().equals(200)){ //保存成功 wmNews.setArticleId((Long)responseResult.getData()); wmNewsMapper.updateById(wmNews); } else{ //保存失败 throw new RuntimeException("保存app端文章失败"); } } private Map<String,List<String>> extractImageAndContent(WmNews wmNews) { //提取文章内容 String content = wmNews.getContent(); List<String> imagesUrls =new ArrayList<>(); List<String> contentTexts =new ArrayList<>(); Map<String,List<String>> scanMaterialsList =new HashMap<>(); List<Map> maps = JSON.parseArray(content, Map.class); //提取文章图片 for (Map map : maps) { if(map.get("type").equals("image")){ String imgUrl = (String) map.get("value"); imagesUrls.add(imgUrl); } if(map.get("type").equals("text")){ String text = (String) map.get("value"); contentTexts.add(text); } } scanMaterialsList.put("imagesUrls",imagesUrls); scanMaterialsList.put("contentTexts",contentTexts); return scanMaterialsList; } } 3.3 启动类扫描feign

调用Feign远程接口时，要在启动类中加入@EnableFeignClients(basePackages = “com.heima.apis”)来对feign的api进行扫描，同时也要引入feign-api模块的依赖

<dependency> <groupId>com.heima</groupId> <artifactId>heima-leadnews-feign-api</artifactId> </dependency> @SpringBootApplication @EnableDiscoveryClient @MapperScan("com.heima.wemedia.mapper") @EnableFeignClients(basePackages = "com.heima.apis") public class WemediaApplication { public static void main(String[] args) { SpringApplication.run(WemediaApplication.class,args); } @Bean public MybatisPlusInterceptor mybatisPlusInterceptor() { MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor(); interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL)); return interceptor; } } 3.4 测试

转到WmNewAutoScanService接口中，ctrl+shift+T创建测试类

@SpringBootTest(classes = WemediaApplication.class) @RunWith(SpringRunner.class) class WmNewAutoScanServiceTest { @Autowired private WmNewAutoScanService wmNewAutoScanService; @Test void autoScanMediaNews() { wmNewAutoScanService.autoScanMediaNews(6236); } } 4 自媒体调用文章微服务feign远程调用服务降级

4.1 feign远程调用服务降级处理的逻辑

4.2 编写降级逻辑

在heima-leadnews-feign-api模块下编写降级逻辑com.heima.apis.article.fallback.IArticleClientFallback类，实现IArticleClient接口

@Component public class IArticleClientFallback implements IArticleClient { @Override public ResponseResult saveArticle(ArticleDto dto) { return ResponseResult.errorResult(AppHttpCodeEnum.SERVER_ERROR,"获取数据失败"); } } 4.3 指定IArticleClient接口指向Feign降级逻辑

@FeignClient(value = "leadnews-article",fallback = IArticleClientFallback.class)

@FeignClient(value = "leadnews-article",fallback = IArticleClientFallback.class) public interface IArticleClient { @PostMapping("/api/v1/article/save") public ResponseResult saveArticle(@RequestBody ArticleDto dto) ; } 4.4 加载feign降级逻辑

因为IArticleClientFallback是在com.heima.apis.article.fallback包下，并不能被spring通过@Component直接加载

因此需要在使用的微服务中加载feign

如使用的微服务是heima-leadnews-wemedia，所以要在com.heima.wemedia.config下创建InitConfig类加载feign降级策略

@Configuration @ComponentScan("com.heima.apis.article.fallback") public class InitConfig { } 4.5 配置降级策略

要么在bootstrap中开启，要么在nacos中实现热更新

这里采用nacos热更新

feign: # 开启feign对hystrix熔断降级的支持 hystrix: enabled: true # 修改调用超时时间 client: config: default: connectTimeout: 2000 readTimeout: 2000

4.6 测试

当前设置超时2s进行降级，测试一下

在com.heima.article.service.impl.ApArticleServiceImpl类中的saveArticle方法添加睡眠3秒进行测试

@Override public ResponseResult saveArticle(ArticleDto dto) { try { Thread.sleep(3000); } catch (InterruptedException e) { throw new RuntimeException(e); } //1.参数检查 if(dto == null){

这次审核6239

@SpringBootTest(classes = WemediaApplication.class) @RunWith(SpringRunner.class) class WmNewAutoScanServiceTest { @Autowired private WmNewAutoScanService wmNewAutoScanService; @Test void autoScanMediaNews() { wmNewAutoScanService.autoScanMediaNews(6239); } }

5 文章审核异步调用

5.1 在自动审核的方法加上@Async注解

Springboot集成异步线程调用

@Override @Async//表明这是一个异步方法 public void autoScanMediaNews(Integer id) { try { Thread.sleep(1000); } catch (InterruptedException e) { throw new RuntimeException(e); } 5.2 在文章发布后调用自动审核方法 //5.审核文章 wmNewAutoScanService.autoScanMediaNews(wmNews.getId()); @Autowired private WmNewAutoScanService wmNewAutoScanService; @Override public ResponseResult submitNews(WmNewsDto wmNewsDto) { // 0.参数检查 if(wmNewsDto == null||wmNewsDto.getContent()==null){ return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID); } //1. 保存或修改文章 WmNews wmNews = new WmNews(); BeanUtils.copyProperties(wmNewsDto,wmNews); //1.1 封面 if(wmNewsDto.getImages()!=null&& wmNewsDto.getImages().size()>0){ String imageStr = StringUtils.join(wmNewsDto.getImages(), ","); wmNews.setImages(imageStr); } //1.2 如果封面为自动-1，则需要手动设置封面规则 if(wmNewsDto.getType().equals(WemediaConstants.WM_NEWS_TYPE_AUTO)){ wmNews.setType(null); } saveOrUpdateWmNews(wmNews); //2.判断是否为草稿，如果为草稿结束当前方法 if(wmNews.getStatus().equals(WmNews.Status.NORMAL.getCode())){ return ResponseResult.okResult(AppHttpCodeEnum.SUCCESS); } //3.不是草稿，保存文章内容与图片素材的关系 //3.1 获取文章内容的图片素材 List<String> imageList=extractUrlInfo(wmNewsDto.getContent()); saveRelativeInfoForContent(imageList,wmNews.getId()); //4.不是草稿，保存文章封面图片与图片素材的关系 saveRelativeInfoForCover(wmNewsDto,wmNews,imageList); //5.审核文章 wmNewAutoScanService.autoScanMediaNews(wmNews.getId()); return ResponseResult.okResult(AppHttpCodeEnum.SUCCESS); } 5.3 在启动类中添加注解开启异步调用

在自媒体引导类中使用@EnableAsync注解开启异步调用

@SpringBootApplication @EnableDiscoveryClient @MapperScan("com.heima.wemedia.mapper") @EnableFeignClients(basePackages = "com.heima.apis") @EnableAsync//开启异步 public class WemediaApplication { public static void main(String[] args) { SpringApplication.run(WemediaApplication.class,args); } @Bean public MybatisPlusInterceptor mybatisPlusInterceptor() { MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor(); interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL)); return interceptor; } } 5.4 综合测试

5.5 使用rabbit MQ来完成异步调用

我的异步调用只要在启动类中加入@EnableAsync就报错，迫不得已采用rabbitMQ

5.5.1 引入依赖

在heima-leadnews-service中引入依赖

<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-amqp</artifactId> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> </dependency> 5.5.2 为微服务配置MQ

在heima-leadnews-article和wemedia的配置文件中添加配置

spring: rabbitmq: host: 192.168.204.129 port: 5672 virtual-host: / username: itcast password: 123321 5.5.3 改造方法，创建监听队列

修改heima-leadnews-wemedia下的com.heima.wemedia.service.impl.WmNewAutoScanServiceImpl类中的autoScanMediaNews方法

@Autowired private RabbitTemplate rabbitTemplate; @Override public void autoScanMediaNews(Integer id) { //1.查询自媒体文章 WmNews wmNews = wmNewsMapper.selectById(id); if (wmNews == null) { throw new RuntimeException("自媒体文章不存在"); } if(wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())){ Map<String,List<String>> scanMaterialsList = extractImageAndContent(wmNews); //2.调用阿里云接口审核文本内容 List<String> contentTexts =scanMaterialsList.get("contentTexts"); boolean isTextScan =true; if(!isTextScan)return; //3.调用阿里云接口审核图片内容 List<String> imagesUrls =scanMaterialsList.get("imagesUrls"); boolean isImageScan =true; if(!isImageScan)return; if(isTextScan && isImageScan) { //审核通过 wmNews.setStatus((short) 9); wmNews.setReason("审核通过"); } } //4.审核成功保存app端的相关文章数据 ArticleDto dto=new ArticleDto(); BeanUtils.copyProperties(wmNews,dto); //布局 dto.setLayout(wmNews.getType()); //频道 dto.setChannelId(wmNews.getChannelId()); //频道名称 WmChannel wmChannel = wmChannelMapper.selectById(wmNews.getChannelId()); if(wmChannel!=null){ dto.setChannelName(wmChannel.getName()); } //作者 dto.setAuthorId(Long.valueOf(wmNews.getUserId())); //作者名称 WmUser wmUser= wmUserMapper.selectById(wmNews.getUserId()); if(wmUser!=null){ dto.setAuthorName(wmUser.getName()); } //设置文章id if(wmNews.getArticleId()!=null){ dto.setId(wmNews.getArticleId()); } dto.setCreatedTime(new Date()); //2.rabbitmq异步处理 Map<String,Object> map=new HashMap<>(); map.put("dto",dto); map.put("wmNewsId",id); rabbitTemplate.convertAndSend("article.queue", map); /*ResponseResult responseResult = iArticleClient.saveArticle(dto); if(responseResult.getCode().equals(200)){ //保存成功 wmNews.setArticleId((Long)responseResult.getData()); wmNewsMapper.updateById(wmNews); } else{ //保存失败 log.error("保存app端文章失败，responseResult: {}", responseResult); throw new RuntimeException("保存app端文章失败"); }*/ } rabbitTemplate.convertAndSend("article.queue", map);

发送到article.queue队列

在heima-leadnews-article模块下创建com.heima.article.mq.ArticleMessageConsumer消费者监听类监听article.queue

@Slf4j @Component public class ArticleMessageConsumer { @Autowired private IArticleClient iArticleClient; @Autowired private RabbitTemplate rabbitTemplate; @RabbitListener(bindings =@QueueBinding( value=@Queue(name="article.queue"), exchange=@Exchange(name="article.direct",type= ExchangeTypes.FANOUT))) public void processMessage(Map<String,Object> map) { ObjectMapper objectMapper = new ObjectMapper(); Object dto = map.get("dto"); Integer id= (Integer) map.get("wmNewsId"); LinkedHashMap<String, Object> linkedHashMap = (LinkedHashMap<String, Object>) dto; ArticleDto articleDto = objectMapper.convertValue(linkedHashMap, ArticleDto.class); // 异步处理文章数据 ResponseResult responseResult = iArticleClient.saveArticle(articleDto); if(responseResult.getCode().equals(200)){ WmNews wmNews = new WmNews(); BeanUtils.copyProperties(dto, wmNews); wmNews.setArticleId((Long)responseResult.getData()); Map<String,Object> params = new HashMap<>(); params.put("id", id); params.put("wmNews", wmNews); params.put("articleId",(Long)responseResult.getData()); rabbitTemplate.convertAndSend("wmNews.queue", params); log.info("发送params成功，param: {}", params); } else{ //保存失败 log.error("保存app端文章失败，responseResult: {}", responseResult); throw new RuntimeException("保存app端文章失败"); } } }

将ResponseResult responseResult = iArticleClient.saveArticle(articleDto);回填的id发到wmNews.queue

在heima-leadnews-wemedia模块下创建com.heima.wemedia.mq.ReceiveWmNewsId消费者监听类监听wmNews.queue

@Component @Slf4j public class ReceiveWmNewsId { @Autowired private WmNewsMapper wmNewsMapper; @RabbitListener(bindings =@QueueBinding( value=@Queue(name="wmNews.queue"), exchange=@Exchange(name="wmNews.direct",type= ExchangeTypes.FANOUT))) public void processMessage(Map<String,Object> map) { ObjectMapper objectMapper = new ObjectMapper(); Integer id= (Integer)map.get("id"); Object wmNews= map.get("wmNews"); Long articleId= (Long)map.get("articleId"); LinkedHashMap<String, Object> linkedHashMap = (LinkedHashMap<String, Object>) wmNews; WmNews articleDto = objectMapper.convertValue(linkedHashMap, WmNews.class); WmNews oldwmNews = wmNewsMapper.selectById(id); BeanUtils.copyProperties(wmNews,oldwmNews); oldwmNews.setStatus((short) 9); oldwmNews.setReason("审核通过"); oldwmNews.setArticleId(articleId); int i = wmNewsMapper.updateById(oldwmNews); if(i == 0){ log.error("更新自媒体文章失败，wmNews: {}", oldwmNews); throw new RuntimeException("更新自媒体文章失败"); } } } 5.5.4 序列化MQ消息

在heima-leadnews-article和wemedia的启动类中添加序列化器

@SpringBootApplication @EnableDiscoveryClient @MapperScan("com.heima.wemedia.mapper") @EnableFeignClients(basePackages = "com.heima.apis") public class WemediaApplication { public static void main(String[] args) { SpringApplication.run(WemediaApplication.class,args); } @Bean public MybatisPlusInterceptor mybatisPlusInterceptor() { MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor(); interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL)); return interceptor; } @Bean public MessageConverter messageConverter(){ return new Jackson2JsonMessageConverter(); } @Bean public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) { RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory); rabbitTemplate.setMessageConverter(messageConverter); return rabbitTemplate; } } @SpringBootApplication @EnableDiscoveryClient @MapperScan("com.heima.article.mapper") public class ArticleApplication { public static void main(String[] args) { SpringApplication.run(ArticleApplication.class,args); } @Bean public MybatisPlusInterceptor mybatisPlusInterceptor() { MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor(); interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL)); return interceptor; } @Bean public MessageConverter messageConverter(){ return new Jackson2JsonMessageConverter(); } @Bean public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) { RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory); rabbitTemplate.setMessageConverter(messageConverter); return rabbitTemplate; } } 5.5.5 加上mq后的综合测试

测试通过在MQ上也检测到消息

6 自管理敏感词过滤

6.1 DFA实现原理

6.2 DFA检索过程

6.3 实现步骤

6.3.1 创建敏感词表

在leadnews-wemedia数据库中到入wm_sensitive.sql

6.3.2 将wm_sensitive对应的实体类和mapper导入 @Data @TableName("wm_sensitive") public class WmSensitive implements Serializable { private static final long serialVersionUID = 1L; /** * 主键 */ @TableId(value = "id", type = IdType.AUTO) private Integer id; /** * 敏感词 */ @TableField("sensitives") private String sensitives; /** * 创建时间 */ @TableField("created_time") private Date createdTime; } @Mapper public interface WmSensitiveMapper extends BaseMapper<WmSensitive> { } 6.3.3 在阿里云接口前自行进行审查

boolean isSensitive= handleSensitiveWords(contentTexts,wmNews);

if(wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())){ Map<String,List<String>> scanMaterialsList = extractImageAndContent(wmNews); //2.调用阿里云接口审核文本内容 List<String> contentTexts =scanMaterialsList.get("contentTexts"); //2.1 敏感词过滤 boolean isSensitive= handleSensitiveWords(contentTexts,wmNews); boolean isTextScan =true; if(!isTextScan)return; //3.调用阿里云接口审核图片内容 List<String> imagesUrls =scanMaterialsList.get("imagesUrls"); boolean isImageScan =true; @Autowired private WmSensitiveMapper wmSensitiveMapper; private boolean handleSensitiveWords(List<String> contentTexts, WmNews wmNews) { boolean isSensitive = true; //1.获取所有敏感词 List<WmSensitive> wmSensitiveList = wmSensitiveMapper.selectList(Wrappers.<WmSensitive>lambdaQuery().select(WmSensitive::getSensitives)); List<String> collect = wmSensitiveList.stream().map(WmSensitive::getSensitives).collect(Collectors.toList()); //2.初始化敏感词库 SensitiveWordUtil.initMap(collect); //3.遍历文章内容查看是否包含敏感词 for(String contentText:contentTexts){ Map<String, Integer> map = SensitiveWordUtil.matchWords(contentText); if(map.size()>0){ //4.如果包含敏感词，修改文章状态 wmNews.setStatus((short) 2); wmNews.setReason("文章内容包含敏感词"); wmNewsMapper.updateById(wmNews); isSensitive = false; break; } } return isSensitive; } 6.3.4 测试

7 图片文字敏感词过滤 7.1 文字图片识别

7.2 Tesseract-OCR

7.3 Tess4j案例

7.3.1 导入依赖

在heima-leadnews-test模块下的tess4j-demo的模块下导入依赖

<dependency> <groupId>net.sourceforge.tess4j</groupId> <artifactId>tess4j</artifactId> <version>4.1.1</version> </dependency> 7.3.2 将训练好的分类器放入资源中

7.3.3 demo

在tess4j-demo的Applcation中

public class Application { /** * 识别图片中的文字 * @param args */ public static void main(String[] args) { // 1.创建Tesseract对象 Tesseract tesseract = new Tesseract(); // 2.设置训练库的位置 tesseract.setDatapath("D:\\Code\\JavaCode\\HeimaToutiao\\heima-leadnews\\heima-leadnews-test\\tess4j-demo\\src\\main\\resources\\tessdata"); // 3.设置识别语言 tesseract.setLanguage("chi_sim"); // 4.设置识别图片 File file = new File("D:\\Code\\JavaCode\\HeimaToutiao\\heima-leadnews\\heima-leadnews-test\\tess4j-demo\\src\\main\\resources\\testdata\\testImage.png"); // 5.识别图片 try { String result = tesseract.doOCR(file); System.out.println(result.replace("\\n|\\r", "")); } catch (TesseractException e) { e.printStackTrace(); } } } 7.3.4 结果

7.4 图片文字敏感词过滤实现

7.4.1 创建工具类

在heima-leadnews-common中创建com.heima.common.tess4j.Tess4jClient工具类，封装tess4j

@Getter @Setter @Component @ConfigurationProperties(prefix = "tess4j") public class Tess4jClient { private String dataPath; private String language; public String doOCR(BufferedImage image) throws TesseractException { //创建Tesseract对象 ITesseract tesseract = new Tesseract(); //设置字体库路径 tesseract.setDatapath(dataPath); //中文识别 tesseract.setLanguage(language); //执行ocr识别 String result = tesseract.doOCR(image); //替换回车和tal键 使结果为一行 result = result.replaceAll("\\r|\\n", "-").replaceAll(" ", ""); return result; } } 7.4.2 工具类被其他微服务使用

想让工具类被其他微服务使用就要拷贝全路径，在当前的resource中的META-INF的spring.factories中添加配置

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\ com.heima.common.exception.ExceptionCatch,\ com.heima.common.aliyun.GreenTextScan,\ com.heima.common.aliyun.GreenImageScan,\ com.heima.common.tess4j.Tess4jClient 7.4.3 在微服务中配置

在heima-leadnews-wemedia中的resource的boostrap.yml中进行配置

tess4j: data-path: D:\Code\JavaCode\HeimaToutiao\heima-leadnews\heima-leadnews-test\tess4j-demo\src\main\resources\tessdata language: chi_sim 7.4.4 添加实现

在WmNewsAutoScanServiceImpl中的handleImageScan方法上添加如下代码

try { for (String image : images) { byte[] bytes = fileStorageService.downLoadFile(image); //图片识别文字审核---begin----- //从byte[]转换为butteredImage ByteArrayInputStream in = new ByteArrayInputStream(bytes); BufferedImage imageFile = ImageIO.read(in); //识别图片的文字 String result = tess4jClient.doOCR(imageFile); //审核是否包含自管理的敏感词 boolean isSensitive = handleSensitiveScan(result, wmNews); if(!isSensitive){ return isSensitive; } //图片识别文字审核---end----- imageList.add(bytes); } }catch (Exception e){ e.printStackTrace(); } 8 静态文件生成

8.1 实现思路

我们在保存/修改文章时就应该同时异步的的生成静态文件，生成静态文件上传到minio中

8.1.1 生成minio接口和实现，并且异步调用

在com.heima.article.service.ArticleFreemarkerService接口

生成静态文件，上传到minio中

public interface ArticleFreemarkerService { /** * 生成静态化页面 * @param apArticle * @param content */ public void buildArticleToMinio(ApArticle apArticle,String content); } @Service @Slf4j @Transactional public class ArticleFreemarkerServiceImpl implements ArticleFreemarkerService { @Autowired private ApArticleContentMapper apArticleContentMapper; @Autowired private Configuration configuration; @Autowired private FileStorageService fileStorageService; @Autowired private ApArticleService apArticleService; /** * 生成静态化页面 * @param apArticle * @param content */ @Async @Override public void buildArticleToMinio(ApArticle apArticle, String content) { if(StringUtils.isNotBlank(content)){ //1.文章内容通过freemarker生成静态html页面 Template template = null; //2 输出流 StringWriter writer = new StringWriter(); try { template = configuration.getTemplate("article.ftl"); //2.1 创建模型 Map<String,Object> contentDataModel=new HashMap(); //content是固定的，因为article.ftl中有<#if content??>${content}</#if> //因为apArticleContent.getContent()获取的是字符串，所以需要转换成对象 contentDataModel.put("content", JSONArray.parseArray(content)); //2.2 合成方法 template.process(contentDataModel,writer); } catch (Exception e) { throw new RuntimeException(e); } //3.把静态页面上传到minio //3.1 文件流 InputStream inputStream = new ByteArrayInputStream(writer.toString().getBytes()); String path = fileStorageService.uploadHtmlFile("",apArticle.getId()+".html",inputStream); //4.把静态页面的路径保存到数据库 apArticleService.update(Wrappers .<ApArticle>lambdaUpdate() .eq(ApArticle::getId,apArticle.getId()) .set(ApArticle::getStaticUrl,path)); } } } 8.1.2 修改saveArticle逻辑

修改com.heima.article.service.impl.ApArticleServiceImpl的saveArticle方法，添加buildArticleToMinio

articleFreemarkerService.buildArticleToMinio(apArticle, dto.getContent()); @Autowired private ApArticleConfigMapper apArticleConfigMapper; @Autowired private ApArticleContentMapper apArticleContentMapper; @Autowired private ArticleFreemarkerService articleFreemarkerService; /** * 保存文章 * @param dto * @return */ @Override public ResponseResult saveArticle(ArticleDto dto) { //1.参数检查 if(dto == null){ return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID); } ApArticle apArticle = new ApArticle(); //org.springframework.beans BeanUtils.copyProperties(dto, apArticle); //2.判断是否存在id if(dto.getId() == null) { //2.1 不存在id ，新增 文章、内容、配置 save(apArticle); //2.1.2 保存文章配置 ApArticleConfig apArticleConfig = new ApArticleConfig(apArticle.getId()); apArticleConfigMapper.insert(apArticleConfig); //2.1.3 保存文章内容 ApArticleContent apArticleContent = new ApArticleContent(); apArticleContent.setArticleId(apArticle.getId()); apArticleContent.setContent(dto.getContent()); apArticleContentMapper.insert(apArticleContent); } else { //2.2 存在id，更新 文章、内容 //2.2.1 更新文章 updateById(apArticle); //2.2.2 更新文章内容 ApArticleContent apArticleContent = apArticleContentMapper.selectOne(Wrappers .<ApArticleContent>lambdaQuery() .eq(ApArticleContent::getArticleId, dto.getId())); apArticleContent.setContent(dto.getContent()); apArticleContentMapper.updateById(apArticleContent); } //异步调用 生成静态文件上传到minio中 articleFreemarkerService.buildArticleToMinio(apArticle, dto.getContent()); //3.返回结果 文章的id return ResponseResult.okResult(apArticle.getId()); } } 8.1.3 开启异步调用

引导类加上@EnableAsyn

@SpringBootApplication @EnableDiscoveryClient @MapperScan("com.heima.article.mapper") @EnableAsync public class ArticleApplication { public static void main(String[] args) { SpringApplication.run(ArticleApplication.class,args); } @Bean public MybatisPlusInterceptor mybatisPlusInterceptor() { MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor(); interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL)); return interceptor; } @Bean public MessageConverter messageConverter(){ return new Jackson2JsonMessageConverter(); } @Bean public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) { RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory); rabbitTemplate.setMessageConverter(messageConverter); return rabbitTemplate; } } 8.1.4 测试

查看minio有没有生成

生成成功，查看数据库，有html生成，说明功能成功